Data Ingestion with Kafka and Kafka Streaming

Learn to use REST Proxy, Kafka Connect, KSQL, and Faust Python Stream Processing and use it to stream public transit statuses using Kafka and Kafka ecosystem to build a stream processing application that shows the status of trains in real-time.

• Introduction to Stream Processing
• In this lesson students will learn what data streaming is. Students will learn the pros and cons of data streaming, and how it compares to traditional data strategies.
• Apache Kafka
• In this lesson we’ll review the architecture and configuration of Apache Kafka.
• Data Schemas and Apache Avro
• This lesson covers data schemas and data schema management, with a focus on Apache Avro.
• Kafka Connect and REST Proxy
• This lesson covers producing and consuming data into Kafka with Kafka Connect and REST Proxy.
• Stream Processing Fundamentals
• Learn to build real-time applications that instantly process events, the concepts of stream processing state storage, windowed processing, and stateful and non-stateful stream processing.
• Stream Processing with Faust
• Students will learn how to use the Python stream processing library Faust to rapidly create powerful stream processing applications.
• KSQL
• Learn how to write simple SQL queries to turn Kafka topics into KSQL streams and tables, and then write those tables back out to Kafka.
• Optimizing Public Transportation
• For your first project, you’ll be streaming public transit status using Kafka and the Kafka ecosystem to build a stream processing application that shows the status of trains in real-time.