Stream Processing Design Patterns with Spark

Learn how to build popular streaming design patterns efficiently with Apache Spark.

Introduction

• Streaming with Spark
• Prerequisites

1. Stream Processing with Spark

• What is stream processing?
• Streaming opportunities and challenges
• Streaming with Apache Spark
• Spark Structured Streaming APIs and SQL
• Setting up the exercise files
• Setting up Kafka
• Setting up MariaDB and Redis

2. Streaming Analytics

• Streaming analytics: Pattern
• Streaming analytics: Use case design
• Streaming analytics: Helper classes
• Streaming analytics: Pipeline implementation
• Streaming analytics: Results review

3. Alerts and Thresholds

• Alerts and thresholds: Pattern
• Alerts and thresholds: Use case design
• Alerts and thresholds: Helper classes
• Alerts and thresholds: Pipeline implementation
• Alerts and thresholds: Review

4. Leaderboards

• Leaderboards: Pattern
• Leaderboards: Use case design
• Leaderboards: Helper classes
• Leaderboards: Pipeline implementation
• Leaderboards: Review

5. Real-Time Predictions

• Real-time predictions: Pattern
• Real-time predictions: Use case design
• Real-time predictions: Helper classes
• Real-time predictions: Pipeline implementation
• Real-time predictions: Review

6. Use Cases

• Use case definition
• Design of the project
• Code walk-through
• Execute and analyze

Conclusion

• Next steps