Big Data Analytics in Near-Real-Time with Apache Kafka Streams

Explore the evolution of ETL processes and learn how to achieve near-real-time data analytics using Apache Kafka streaming in this comprehensive conference talk. Dive into the challenges of managing and distributing large volumes of data from multiple sources, and discover why traditional ETL methods are becoming less effective. Compare standard batch ETL processes with streaming data techniques, and gain hands-on experience processing data in real-time using Apache Kafka as the shared backbone. Witness demonstrations of real-time data aggregations and horizontal scaling through a combination of Kafka, Kafka Connect, KSQL, and Kafka Streams. Connect streaming data with SQL Server and ElasticSearch, and understand the benefits of instant access to processed data as it arrives. By the end of this talk, grasp the potential of living in a world free from waiting for batch processes to complete, and embrace the future of data analytics.

Intro
Donuts
Demo
What most people know
Whats wrong
Apache Kafka
Why use Kafka
Demo setup
Demo start
Create streaming database
Kafka setup
Kafka Connect
Auto Offset
Kafka Control Center
Aggregation
Recap
Push to multiple places
One caveat
Kotlin
Kafka Broker