Preventing Common Pitfalls in Production Streaming Jobs

Explore critical aspects of running streaming jobs in production environments through this 54-minute conference talk by Databricks. Learn how to prevent common pitfalls that can cause serious issues when productionizing streaming jobs. Dive into four key topics: configuring input parameters to handle unexpected data volume increases, tuning stateful streaming parameters to avoid infinite state accumulation, optimizing Structure Streaming output parameters to prevent small file problems, and modifying streaming jobs in production with checkpoints. Gain practical, hands-on examples of issue manifestation and prevention techniques. Equip yourself with the knowledge to design performant and fault-tolerant streams, ensuring smooth operation in production environments.

Introduction
Agenda
Input Parameters
Tuning Max Files per Trigger
Tuning State Parameters
Performing Aggregates
Watermark
Help Function
State Store Provider
State Store Limits
Delta Back State
Delta Back State Code
Performance
Output Parameters
Small Things to Consider