Running Apache Kafka in Production - 6 Key Recommendations

Discover essential recommendations for running Apache Kafka in production from Jun Rao, one of its original creators and ongoing contributors. Learn how to optimize your Kafka setup by focusing on operational aspects, serialization and schema management, utilizing Kafka as a central nervous system, implementing dead letter queues, understanding compacted topics, and exploring new use cases enabled by Kafka's cloud evolution. Gain valuable insights on maximizing Kafka's potential in production environments, including tips on monitoring, alerting, and logging for distributed systems, choosing the right serialization mechanism, and leveraging cloud capabilities for unlimited storage and engineering efficiency.

- Intro
- Nail down the operational part
- Reason properly about serialization and schemas up front
- Utilize Dead Letter Queues
- Use Kafka as a central nervous system vs. a single cluster
- Understand compacted topics
- Imagine new use cases enabled by Kafka's recent evolution
- It's a wrap!