Learn Apache Kafka for Beginners

Learn the Apache Kafka basics. Learn how to use this enterprise-scale data streaming technology to read and write data from multiple sources—without managing multiple integrations.

Introduction

• Kafka course introduction
• Apache Kafka in five minutes
• Course objectives

1. Kafka Theory

• Topics, partitions, and offsets
• Producers and message keys
• Consumers and deserialization
• Consumer groups and consumer offsets
• Brokers and topics
• Topic replication
• Producer acknowledgments and topic durability
• ZooKeeper
• Kafka KRaft: Removing ZooKeeper
• Theory roundup

2. Starting Kafka

• Important: Starting Kafka and lectures order
• Starting Kafka with Conduktor: Multi-platform
• macOS X: Download and set up Kafka in PATH
• macOS X: Start ZooKeeper and Kafka
• macOS X: Using brew
• Linux: Download and set up Kafka in PATH
• Linux: Start ZooKeeper and Kafka
• Windows WSL2: Download Kafka and PATH setup
• Windows WSL2: Start ZooKeeper and Kafka
• Windows WSL2: How to fix problems
• Windows non-WSL2: Start ZooKeeper and Kafka

3. Starting Kafka without ZooKeeper

• macOS X: Start Kafka in KRaft mode
• Linux: Start Kafka in KRaft mode
• Windows WSL2: Start Kafka in KRaft mode

4. CLI (Command-Line Interface) 101

• CLI introduction
• Kafka topics CLI
• Kafka console producer CLI
• Kafka console consumer CLI
• Kafka consumers in groups
• Kafka consumer groups CLI
• Resetting offsets

5. Kafka UI

• Conduktor: Demo

6. Kafka Java Programming 101

• Kafka SDK list
• Creating a Kafka project
• Java producer
• Java producer callbacks
• Java producer with keys
• Java consumer
• Java consumer: Graceful shutdown
• Java consumer inside consumer group
• Java consumer incremental cooperative rebalance and static group membership
• Java consumer incremental cooperative rebalance: Practice
• Java consumer auto offset commit behavior
• Programming: Advanced tutorials

7. Kafka Real-World Project

• Real-world project overview

8. Kafka Wikimedia Producer and Advanced Producer Configurations

• Wikimedia producer project setup
• Wikimedia producer implementation
• Wikimedia producer run
• Wikimedia producer: Producer config intros
• Producer acknowledgments deep dive
• Producer retries
• Idempotent producer
• Safe Kafka producer settings
• Wikimedia producer safe producer implementation
• Kafka message compression
• linger.ms and batch.size producer settings
• Wikimedia producer high-throughput implementation
• Producer default partitioner and sticky partitioner
• [Advanced] max.block.ms and buffer.memory

9. OpenSearch Consumer and Advanced Consumer Configurations

• OpenSearch consumer: Project overview
• OpenSearch consumer: Project setup
• Setting up OpenSearch on Docker
• Setting up OpenSearch on the Cloud
• OpenSearch 101
• OpenSearch consumer implementation: Part 1
• OpenSearch consumer implementation: Part 2
• Consumer delivery semantics
• OpenSearch consumer implementation: Part 3 idempotence
• Consumer offsets commit strategies
• OpenSearch consumer implementation: Part 4 delivery semantics
• OpenSearch consumer implementation: Part 5 batching data
• Consumer offset reset behavior
• OpenSearch consumer implementation: Part 6 replaying data
• Consumer internal threads
• Consumer replica fetching: Rack awareness

10. Kafka Extended APIs for Developers

• Kafka extended APIs: Overview
• Kafka Connect introduction
• Kafka Connect Wikimedia and Elasticsearch hands-on
• Kafka Streams introduction
• Kafka Streams hands-on
• Kafka Schema Registry introduction
• Kafka Schema Registry hands-on
• Which Kafka API should you use?

11. Real-World Insights and Case Studies: Big Data and Fast Data

• Choosing partition count and replication factor
• Kafka topics naming convention
• Case study: MovieFlix
• Case study: GetTaxi
• Case study: MySocialMedia
• Case study: MyBank
• Case study: Big data ingestion
• Case study: Logging and metrics aggregation

12. Kafka in the Enterprise for Admins

• Kafka cluster setup high-level architecture overview
• Kafka monitoring and operations
• Kafka security
• Kafka multi-cluster and MirrorMaker
• Advertised listeners: Kafka client and server communication protocol

13. Advanced Topics Configurations

• Changing a topic configuration
• Segment and indexes
• Log cleanup policies
• Log cleanup delete
• Log compaction theory
• Log compaction practice
• Unclean leader election
• Large messages in Kafka

Conclusion

• What's next?