Chasing the Stream Processing Utopia

Explore the challenges and advancements in real-time data processing at internet scale in this conference talk from Strange Loop. Dive into the world of stream processing as Kartik Paramasivam shares LinkedIn's experience handling trillions of events daily using Apache Kafka and Samza. Learn about the evolution from batch to stream processing, the importance of real-time data reactions in social computing, and the current state of stream processing technology. Discover techniques for efficient event processing, including application local state for improved performance, and the need for multiple programming languages in expressing processing logic. Gain insights into scaling ingestion, managing Kafka clusters, and addressing bottlenecks in stream processing. Examine concepts such as stateless and stateful applications, windowing, event time-based processing, and exactly once processing. Understand the challenges of maintaining temporary state, working with batch sources, and the tools ecosystem surrounding stream processing technologies.

Intro
Stream Processing Technologies
Types of Stream Processing
Stateless Application: Simple Filtering
Stateless Application: Data Enrichment
Stateful Application: Aggregation
Windowing
Event Time-Based Processing
Accuracy
Exactly Once Processing - Ingestion
Exactly Once Processing - Pipelined processing
Scaling Ingestion
Running Kafka @ Scale
Kafka Cluster Management Woes Large deployment
Kafka Cruise Control 1
Cruise Control Architecture
Scaling Processing: Challenges
Typical Bottlenecks in Stream Processing
Accessing Adjunct Data - Using Remote DB
Accessing Adjunct Data - Using Local DB
Maintaining Temporary State : Incremental Checkpoints
Local State Gotchas!
And Batch Sources..
Stream Application in Batch
Apache Beam
Tools Ecosystem