Tracing Real-Time Distributed Systems

Explore the challenges and solutions of tracing real-time distributed systems in this 41-minute conference talk from SREcon19 Europe/Middle East/Africa. Delve into the complexities of applying distributed tracing concepts to latency-sensitive, real-time data streaming in large-scale trading systems. Learn how to model asynchronous message flows in multi-service pipelines, handle long-running sessions, manage fan-in/fan-out data flows, and reduce storage costs while maintaining granular in-process tracing data. Discover techniques for diagnosing issues, measuring service level indicators, and improving observability in client-side applications. Gain insights into the Bloomberg LP approach to tracing, including practical examples from their trading system architecture and implementation strategies for effective system monitoring and debugging.

Intro
Overview
What is the Bloomberg Terminal?
About Bloomberg
Example - A Search Engine
(Possible) Architecture View
Trace Concepts. Span Model
Tracing Architecture (One Interpretation)
Using Trace • Triage Debugging
Tracing Model & Implementation
A Trading System
Architecture - Simple View
System Characteristics
How we traced it?
What does it look like?
The interesting bit
What we do with it?
Message Fan-Out (broadcast)
#2 Splitting Messages
Message Conflation
What's Next?
We are hiring!