Isolating Streaming Ingest and Queries Using RocksDB

Explore an innovative real-time analytics architecture implemented in the Rockset database using RocksDB to effectively isolate streaming data ingestion from query serving. Dive into the challenges of compute contention in real-time analytics systems and examine common approaches like scaling, replication, and querying from shared storage. Learn about the separation of compute and storage, the use of cloud object storage for durability, and SSD as a shared hot storage tier for low-latency reads. Discover how a custom-built RocksDB memtable replicator propagates changes to remote instances in milliseconds, enabling compute isolation while allowing queries on the most recent data. Gain insights into this architecture that addresses the problem of competing functions contending for available compute resources in real-time analytics systems.

Isolating Streaming Ingest and Queries Using RocksDB