solves the key challenges in real-time analytics:
- High ingest throughput
- Low-latency ingestion
- Fast query performance
- Efficient handling of data updates and late-arriving data
- Streamlined data management
- Fast ingest with : new data is initially written to the , which is optimized for high-speed inserts and updates. This process ensures that real-time applications easily handle rapid streams of incoming data. Mutability—upserts, updates, and deletes happen seamlessly.
- Efficient analytics with : as the data cools and becomes more suited for analytics, it is automatically converted to the . This columnar format enables fast scanning and aggregation, optimizing performance for analytical workloads while also saving significant storage space.
- Faster queries on compressed data in : in the conversion, chunks are compressed by up to 98%, and organized for efficient, large-scale queries. Combined with chunk skipping, this helps you save on storage costs and keeps your queries operating at lightning speed.
- Fast modification of compressed data in : just use SQL to add or modify data in the . is optimized for superfast INSERT and UPSERT performance.
- Full mutability with transactional semantics: regardless of where data is stored, provides full ACID support. Like in a vanilla database, inserts and updates to the and are always consistent, and available to queries as soon as they are completed.
Columnar batches
uses columnar collocation and columnar compression within row-based storage to optimize analytical query performance while maintaining full compatibility. This approach ensures efficient storage, high compression ratios, and rapid query execution.
A chunk is converted to a chunk by successfully grouping together sets of rows (typically up to
1000) into a single batch, then converting the batch into columnar form.
Each compressed batch does the following:
- Encapsulates columnar data in compressed arrays of up to 1,000 values per column, stored as a single entry in the underlying compressed table
- Uses a column-major format within the batch, enabling efficient scans by co-locating values of the same column and allowing the selection of individual columns without reading the entire batch
- Applies advanced compression techniques at the column level, including run-length encoding, delta encoding, and Gorilla compression, to significantly reduce storage footprint (by up to 95%) and improve I/O performance.
Segmenting and ordering data
To optimize query performance, allows explicit control over how data is physically organized within columnar storage. By structuring data effectively, queries can minimize disk reads and execute more efficiently, using vectorized execution for parallel batch processing where possible.
- Group related data together to improve scan efficiency: organizing rows into logical segments ensures that queries
filtering by a specific value only scan relevant data sections. For example, in the above, querying for a specific ID
is particularly fast. (Implemented with
SEGMENTBY.) - Sort data within segments to accelerate range queries: defining a consistent order reduces the need for post-query
sorting, making time-based queries and range scans more efficient. (Implemented with
ORDERBY.) - Reduce disk reads and maximize vectorized execution: a well-structured storage layout enables efficient batch processing (Single Instruction, Multiple Data, or SIMD vectorization) and parallel execution, optimizing query performance.
Data mutability
Traditional databases force a trade-off between fast updates and efficient analytics. Fully immutable storage is impractical in real-world applications, where data needs to change. Asynchronous mutability—where updates only become visible after batch processing—introduces delays that break real-time workflows. In-place mutability, while theoretically ideal, is prohibitively slow in columnar storage, requiring costly decompression, segmentation, ordering, and recompression cycles. Hypercore navigates these trade-offs with a hybrid approach that enables immediate updates without modifying compressed data in place. By staging changes in an interim chunk, hypercore allows updates and deletes to happen efficiently while preserving the analytical performance of columnar storage.
Real-time writes without delays
All new data which is destined for a chunk is first written to an interim chunk, ensuring high-speed ingestion and immediate queryability. Unlike fully columnar systems that require ingestion to go through compression pipelines, hypercore allows fresh data to remain in a fast row-based structure before being later compressed into columnar format in ordered batches as normal. Queries transparently access both the and chunks, meaning applications always see the latest data instantly, regardless of its storage format.Efficient updates and deletes without performance penalties
When modifying or deleting existing data, hypercore avoids the inefficiencies of both asynchronous updates and in-place modifications. Instead of modifying compressed storage directly, affected batches are decompressed and staged in the interim chunk, where changes are applied immediately. These modified batches remain in row storage until they are recompressed and reintegrated into the (which happens automatically via a background process). This approach ensures updates are immediately visible, but without the expensive overhead of decompressing and rewriting entire chunks. This approach avoids:- The rigidity of immutable storage, which requires workarounds like versioning or copy-on-write strategies
- The delays of asynchronous updates, where modified data is only visible after batch processing
- The performance hit of in-place mutability, which makes compressed storage prohibitively slow for frequent updates
- The restrictions some databases have on not altering the segmentation or ordering keys