Demystifying Stream Processing with Apache Kafka

Explore the fundamentals of stream processing and Apache Kafka in this 55-minute conference talk from Philly ETE 2016. Delve into the core features of stream processing frameworks, including scalability, fault tolerance, and event processing order guarantees. Learn how to map practical data problems to stream processing and write applications that process data streams at scale. Discover Kafka's new stream processing library, Kafka Streams, and understand its unique design decisions and tradeoffs. Gain insights into Kafka Streams' low-overhead development approach, its integration with existing deployment tools, and how it leverages Kafka's features for scalability and fault tolerance. Cover topics such as partitioning messages, batch vs. stream processing, the Lambda Architecture, and the Kafka Processor API. Examine the execution model, load balancing, state storage, and time handling in Kafka applications. By the end of this talk, grasp the key concepts of stream processing and how Kafka Streams represents a new design point in the stream processing landscape.

Intro
Myths about Stream Processing
Summary of Stream Processing
Stream Processing Challenges
Lambda Architecture
What is Kafka
Partitioning messages
Batch vs Stream processing
Stream processing as a consumer
Tapas Trains
Library vs Framework
Kafka Processor API
Kafka Application
Execution Model
Load Balancing
State Storage
Map to Physical Processes
Time
Kafka Connect