A Kafka Client's Request - There and Back Again

Explore the intricate journey of data through Apache Kafka® in this 40-minute Devoxx conference talk. Delve into the complex process behind writing to and reading from Kafka, following a request from the initial producer.send() or consumer.poll() call to disk storage and back to the client. Examine client and broker configurations affecting request handling, and learn about crucial metrics for monitoring each stage of the request life cycle. Gain in-depth knowledge of read and write request mechanics, enhancing your ability to debug and analyze Kafka performance. Cover topics including schema and topic creation, partitioning configurations, batching settings and metrics, request configurations and general metrics, network threads, request queues, I/O threads, page cache/disk configuration, and the return journey to the producer. Equip yourself with the expertise to navigate Kafka's intricate data flow, making future troubleshooting and optimization tasks more manageable.

Intro
A Schema and a Topic
Partitioning Configurations
Batching Configurations
Batching Metrics
Request Configurations
Request Metrics (General)
Socket Receive Buffer
Network threads
Request Queue
I/O Threads, aka Request Handler Threads
Page Cache/Disk Configuration and Monitoring
Response Queue
Network thread hand-off
Socket Send Buffer
And back again to the Producer
Start your own journey..