Network Observability at LinkedIn - Scaling Flow Data Processing with Erlang

Explore the intricacies of network observability at LinkedIn in this conference talk from Code BEAM America 2022. Delve into the challenges and solutions of managing network flow data at massive scale, with LinkedIn's infrastructure handling millions of requests per second. Learn about the Erlang-based system developed for collecting, processing, and storing network flow data, exporting flows at a rate of 2 million packets per second. Gain insights into the system's architecture, scaling challenges, and applications of flow data. Discover how LinkedIn maintains the health of its vast infrastructure through innovative data collection and processing techniques. Ideal for Erlang developers, observability engineers, and backend engineers interested in high-volume data processing systems.

00:00 - - Intro and Agenda
02:03 - - Intro to InFlow
07:16 - - Applications of Flow Data
11:16 - - InFlow Collector Architecture
22:19 - - InFlow Collector at Scale