Streaming Data Pipelines on Apache Mesos - Lessons Learned

Explore streaming data pipelines on Apache Mesos in this 42-minute conference talk by Dean Wampler, Ph.D. from Lightbend. Gain insights into the Lightbend Fast Data Platform, a suite of open-source streaming tools running on Mesosphere DC/OS. Discover why Mesos was chosen for its flexibility in supporting diverse requirements of streaming and batch data processing. Learn about the key Mesos features crucial for Lightbend's needs, illustrated through examples of how tools like Kafka, Spark, Flink, Akka Streams, and Kafka Streams work with Mesos. Delve into lessons learned from running applications on Mesos and get a glimpse of future developments. Cover topics including Hadoop's limitations, streaming services, microservices, Spark Streaming, continuous processing, scalability, checkpointing, dynamic resource allocation, machine learning, interoperability, and various streaming tools. Understand the relationship between microservices and Fast Data, exploring common mechanisms and architectures in this informative talk by a big data expert.

Intro
Why do we care
Should we use Hadoop
Whats wrong with Hadoop
Whats not to like
Yarn
Container Support
Streaming Services
Microservices
Why Mesos
Spark
Streaming
Continuous Processing
Lifespan
Scalability
Checkpointing
Dynamic Resource Allocation
Latency Budget
Machine Learning
Interoperability
Tools
Kafka streams
Akka streams
Embedding Kafka and Akka streams
Spark Streaming
Flink
Microservices and Fast Data
Microservice Architecture
Streaming Architectures
Microservices vs Big Data
Common Mechanisms
Wheres the Beer