Apache Flink: Batch Mode Data Engineering

Explore how to build batch mode data pipelines with Apache Flink, the powerful and popular stream-processing platform.

Introduction

• Batch mode engineering

1. Apache Flink

• What is Apache Flink?
• Apache Flink features
• Architecture of Apache Flink
• Flink program structure
• Flink execution flow

2. Setting Up Flink

• Installing Flink standalone
• Creating a Flink project
• Build a sample Flink program
• Running jobs on the cluster
• Using the Flink web interface
• Setting up the exercise files

3. Dataset API

• DataSet API concepts
• Reading a CSV File
• Using Map
• Using FlatMap
• Using filters
• Using aggregates
• Using Reduce

4. Advanced Capabilities

• Using POJO classes
• Join operations
• Using MySQL with Flink
• Using broadcast variables

5. Use Case Project

• Problem definition
• Computing total score
• Printing scores for physics
• Computing average scores across subjects
• Find the top student for each subject

Conclusion

• Next steps