EoS in Kafka - Listen Up, I Will Only Say This Once

Explore the intricacies of exactly-once semantics (EoS) in Apache Kafka in this comprehensive conference talk from Strange Loop. Dive deep into the recent additions to Kafka that enable EoS, including idempotence and transactions in Kafka clients. Gain insights into the specific semantics of Kafka distributed transactions and the underlying mechanics that allow them to scale efficiently. Learn about Kafka's unique approach to standard two-phase commit protocols, the maintenance and replication of transaction state, and how various failure scenarios are handled. Discover the future improvements planned to enhance exactly-once stream processing with Kafka. Led by Jason Gustafson, a top contributor to Kafka and part of the team that designed and implemented EoS support, this talk covers topics such as Kafka producers and consumers, processing models and semantics, idempotent producers, Chandy-Lamport snapshots, transaction log availability, zombie fencing, consumer read isolation, broker and consumer fetch handling, concurrent transactions, performance observations, and end-to-end EoS.

Intro
KAFKA IN A HURRY
KAFKA PRODUCER
PRODUCER PROTOCOL
KAFKA CONSUMER
CONSUMER PROTOCOL
PROCESSING MODEL
PROCESSING SEMANTICS
AT-LEAST-ONCE SEMANTICS
SEMANTIC WEAKNESSES
IDEMPOTENT PRODUCER
CHANDY-LAMPORT SNAPSHOTS
KAFKA TRANSACTIONS
TRANSACTION LOG AVAILABILITY
ZOMBIE FENCING
CONSUMER READ ISOLATION
BROKER FETCH HANDLING
CONSUMER FETCH HANDLING
CONCURRENT TRANSACTIONS
KAFKA EXACTLY ONCE SEMANTICS
PERFORMANCE OBSERVATIONS
END-TO-END EOS
FUTURE IMPROVEMENTS
EOS TEAM
APACHE KAFKA