Optimizing Amazon EventBridge Pipes with Apache Kafka for ProSiebenSat.1

Explore the optimization of Amazon EventBridge Pipes with Apache Kafka in this conference talk from GOTO EDA Day 2024. Learn how ProSiebenSat.1 Media SE tackled the challenge of serving millions of customers during major events for their streaming application, Joyn. Discover the journey of optimizing Kafka within their system, including the decision to use Confluent Kafka over Amazon MSK. Gain insights into integrating Kafka with Amazon EventBridge Pipes, addressing limitations and issues encountered. Examine architectural patterns like Claim-check and Point-to-point fan-out for enhancing resilience and efficiency. Understand the speakers' wishlist for AWS improvements in payload size, monitoring capabilities, and processing speed. Conclude with valuable lessons learned and strategies for effectively leveraging Kafka in complex systems.

Intro
The challenge
Why Kafka?
EventBridge with Kafka
EventBridge with Kafka alternative
How we did this
Implemented solution
AWS wishes
Summary
Outro