The Evolution of a Planetary-scale Distributed Database

Explore the evolution of New Relic's in-house telemetry database (NRDB) in this 37-minute conference talk from Strange Loop 2022. Dive into the journey from a pre-Docker era to a globally distributed, cloud-agnostic system orchestrated with Kubernetes. Learn about the challenges faced and solutions implemented in building and operating a planetary-scale distributed database. Gain insights into the theory and practice of large-scale distributed systems, including concepts like decoupling ingest and query operations, cell architecture, cloud migration, and metadata management. Discover how NRDB handles exabytes of data across continuously rebalanced clusters while providing a seamless user experience. Benefit from the speaker's 20+ years of experience in building distributed systems for search, eCommerce, and observability domains.

Intro
Outline
Conceptual Architectural Design
Product Design Goals
In the beginning
Decoupling Ingest & Query, v1
Ingest & Query Routing, v1
Cell Architecture & Cloud Migration
Ingest & Query Routing, v2
Problem: Bin-packing is (NP-)Hard
Decoupling Storage
Decoupling Ingest & Query, v2
Regions & Cloud Providers
Your customer doesn't care about your architecture
Decoupling Ingest & Query, v3
Decoupling Metadata
Differentiated Query Cells