Demystifying Kafka

Explore the intricacies of Apache Kafka in this 42-minute conference talk from Devoxx. Delve into essential concepts such as partition rebalancing, offset management, replication, producer/broker request fetching, and file segments to understand Kafka's scalability, resilience, performance, and fault tolerance. Learn about Kafka transactions and their applications, while gaining insights into potential pitfalls when deploying Kafka in production environments. Discover how Kafka can be utilized for building tracking systems, messaging platforms, high-performance streaming systems, real-time analysis, and audit logs. Gain valuable knowledge from Senior Software Engineer Nakul Mishra's experience in developing a scalable event-store and messaging platform capable of storing billions of messages.

Intro
Starting with a simple idea
Three challenges
Ordering quarantees, traditional messaging system
Log Structure
Parallel Consumption - comes from Log
Resiliency - comes from Replication
Kafka & Event sourcing?
Katka - for event sourcing ?
What's missing
Kalka, a database inside out
Identity & concurrency control
Aggregates & Projections
Kalka Streams
What we learned in the process?