How Uber Audits Real-Time Infrastructure of Trillion+ Messages

Explore how Uber audits its real-time infrastructure handling over a trillion messages daily in this 41-minute Devoxx conference talk. Dive into the motivation, design, and implementation challenges behind Chaperone, Uber's open-source solution for measuring completeness and reliability at massive scale. Learn about use cases in Uber Eats and fraud detection, the Uber ecosystem, and Kafka lifecycle. Discover how Chaperone addresses ordered messaging, latency, and metrics across multiple data centers. Gain insights into the technical components, including Zookeeper and Cassandra, and understand the prototyping process. Presented by Ankur Bansal, a senior engineer in Uber's Streaming team and Apache Kylin committer, this talk offers valuable lessons for scaling distributed systems and cloud infrastructure.

Intro
Agenda
Use Cases
Uber Eats
Fraud Detection
Realtime Processing
Trillion Messages
What Kind of Data
Uber Ecosystem
SLA
Kafka Life Cycle
Why Audit
Counting Messages
Ordered Messaging
chaperone service
query results
metrics
latency
Chaperon
Solution
Zookeeper
Cassandra
Comparison
Summary
Prototypes
Questions