Apache Flink: Real-Time Data Engineering

Discover how to build a real-time stream processing pipeline with Apache Fink. Learn about the platform's windowing, event-time processing, and state management features.

Introduction

• Real-time processing and analytics

1. Apache Flink

• What is Apache Flink?
• Streaming with Apache Flink
• DataStream API
• Related prerequisite courses
• Setting up exercise files

2. DataStream API

• Setting up the Flink environment
• Reading from a stream source
• Processing streaming data
• Writing to a stream sink
• Using keyed streams
• ProcessFunction
• Splitting a stream
• Merging multiple streams

3. Windowing

• Windowing concepts
• Using a Kafka streaming source
• Using sliding windows
• Using session windows
• Window joins

4. Event Time Processing

• Time attributes in Flink
• Watermarks
• Setting up event time
• Processing with event time
• Writing to a Kafka sink

5. State Management

• State management in Flink
• Defining states
• Using states
• Advanced state management

6. Use Case Project

• Problem definition
• Computing summary counts
• Computing activity durations

Conclusion

• Next steps