Distributed Real Time Stream Processing - Why and How

Explore distributed real-time stream processing frameworks in this 42-minute conference talk from Scala Days New York 2016. Dive into popular open-source solutions like Spark Streaming, Storm, Samza, and Flink, comparing their similarities, differences, and trade-offs. Gain insights into theoretical foundations, common pitfalls, and popular architectures for handling the increasing demand for fast processing of immense data from disparate sources. Learn how to choose the right framework for various use cases, including trading, social networks, Internet of Things, and system monitoring. Discover comprehensive overviews of modern streaming solutions, runtime and programming models, fault tolerance, state management, and performance considerations. Examine project maturity for different frameworks and receive general guidelines and recommendations for implementing streaming solutions.

Intro
The Data Deluge
Distributed Stream Processing
Points of interest
Runtime and Programming Model
Native Streaming
Micro-batching
Apache Streaming Landscape
System Comparison
Fault Tolerance
Managing State
Counting Words Revisited
Performance
Project Maturity [Storm & Trident]
Project Maturity [Spark Streaming]
Project Maturity [Samza]
Project Maturity [Flink]
Summary
General Guidelines
Recommendations [Storm & Trident]
Recommendations [Spark Streaming]
Recommendations Samza
Recommendations [Apex]
Recommendations [Flink]
Dataflow and Apache Beam
Questions