Hypercore

Reference information about the TimescaleDB hybrid row-columnar storage engine

Hypercore is a hybrid row-columnar storage engine in TimescaleDB. It is designed specifically for real-time analytics and powered by time-series data. The advantage of hypercore is its ability to seamlessly switch between row-oriented and column-oriented storage, delivering the best of both worlds:

Hypercore solves the key challenges in real-time analytics:

• High ingest throughput
• Low-latency ingestion
• Fast query performance
• Efficient handling of data updates and late-arriving data
• Streamlined data management

Hypercore‘s hybrid approach combines the benefits of row-oriented and column-oriented formats:

• Fast ingest with rowstore: new data is initially written to the rowstore, which is optimized for high-speed inserts and updates. This process ensures that real-time applications easily handle rapid streams of incoming data. Mutability—upserts, updates, and deletes happen seamlessly.

• Efficient analytics with columnstore: as the data cools and becomes more suited for analytics, it is automatically converted to the columnstore. This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space.

• Faster queries on compressed data in columnstore: in the columnstore conversion, hypertable chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with chunk skipping, this helps you save on storage costs and keeps your queries operating at lightning speed.

• Fast modification of compressed data in columnstore: just use SQL to add or modify data in the columnstore. TimescaleDB is optimized for superfast INSERT and UPSERT performance.

• Full mutability with transactional semantics: regardless of where data is stored, hypercore provides full ACID support. Like in a vanilla Postgres database, inserts and updates to the rowstore and columnstore are always consistent, and available to queries as soon as they are completed.

For an in-depth explanation of how hypertables and hypercore work, see the Data model.

Samples

Best practice for using hypercore is to:

1. Enable columnstore on a hypertable

   For efficient queries, remember to segmentby the column you will use most often to filter your data. For example:

   • Hypertables:

     Use CREATE TABLE:

     CREATE TABLE crypto_ticks (
       "time" TIMESTAMPTZ,
       symbol TEXT,
       price DOUBLE PRECISION,
       day_volume NUMERIC
     ) WITH (
       timescaledb.hypertable,
       timescaledb.segmentby='symbol',
       timescaledb.orderby='time DESC'
     );
     

     For TimescaleDB v2.23.0 and higher, the table is automatically partitioned on the first column in the table with a timestamp data type. If multiple columns are suitable candidates as a partitioning column, TimescaleDB throws an error and asks for an explicit definition. For earlier versions, set partition_column to a time column.

     If you are self-hosting TimescaleDB v2.20.0 to v2.22.1, to convert your data to the columnstore after a specific time interval, you have to call add_columnstore_policy after you call CREATE TABLE

     If you are self-hosting TimescaleDB v2.19.3 and below, create a Postgres relational table, then convert it using create_hypertable. You then enable hypercore with a call to ALTER TABLE.

   • Continuous aggregates:

   1. Use ALTER MATERIALIZED VIEW for a continuous aggregate:

      ALTER MATERIALIZED VIEW assets_candlestick_daily set (
         timescaledb.enable_columnstore = true,
         timescaledb.segmentby = 'symbol');
      

   2. Create a columnstore_policy that automatically converts chunks in a hypertable to the columnstore at a specific time interval. For example:

      CALL add_columnstore_policy('assets_candlestick_daily', after => INTERVAL '1d');
      

   TimescaleDB is optimized for fast updates on compressed data in the columnstore. To modify data in the columnstore, use standard SQL.

2. View the policies that you set or the policies that already exist

   SELECT * FROM timescaledb_information.jobs
   WHERE proc_name='policy_compression';
   

   See timescaledb_information.jobs.

You can also convert_to_columnstore and convert_to_rowstore manually for more fine-grained control over your data.

Limitations

chunks in the columnstore have the following limitations:

• ROW LEVEL SECURITY is not supported on chunks in the columnstore.

Available functions

Policies

• add_columnstore_policy(): set a policy to automatically move chunks in a hypertable to the columnstore when they reach a given age
• remove_columnstore_policy(): remove a columnstore policy from a hypertable

Configuration

• ALTER TABLE (hypercore): enable the columnstore for a hypertable

Manual conversion

• convert_to_columnstore(): manually add a chunk to the columnstore
• convert_to_rowstore(): move a chunk from the columnstore to the rowstore

Statistics and information

• chunk_columnstore_stats(): get statistics about chunks in the columnstore
• hypertable_columnstore_stats(): get columnstore statistics related to the

columnstore

• timescaledb_information.chunk_columnstore_settings: get information about settings on each chunk in the columnstore
• timescaledb_information.hypertable_columnstore_settings: get information about columnstore settings for all hypertables