The Magical Rebalance Protocol of Apache Kafka

Explore the inner workings of Apache Kafka's hidden protocol for managing work assignments between multiple client application instances in this 41-minute Strange Loop Conference talk. Dive deep into the binary network protocol and Java APIs of Kafka's little-known assignment protocol, understanding how it facilitates leader election and task assignment. Learn how Kafka Consumers, Connect, and Streams API utilize this protocol for task management. Discover how to extend the protocol to implement custom task assignment algorithms in your applications, even without using Kafka for other purposes. Gain insights into partitions, replication, consumer groups, and the intricacies of joining groups, selecting assignment protocols, and handling rebalancing. Explore future plans and references for further understanding of this magical rebalance protocol.

Intro
Partitions
Replication
Kafka Consumer
Multiple Consumers
Consumer Groups
Requirements
Whats a protocol to do?
Layered protocol
How the protocol is used today?
Dramatis Personae
Who is the Coordinator
Joining Group
Selecting an Assignment Protocol
Kafka Protocol
Sync Group
Heartbeat
Leave Group
Rebalancing
Schema Registry
Stream-Stream Join
To Summarize!
Some future plans...
References