Delta Lake Streaming - Internals and Query Progress Logs

Dive deep into the internals of Structured Streaming with Delta Lake in this 29-minute technical talk from Databricks. Explore the seamless integration of Delta Lake and structured streaming for real-time data processing capabilities. Understand the functional components of structured streaming using Delta as a source, including Query Progress Logs (QPL) and their importance in production environments. Learn how to track streaming job progress and map it to source Delta tables using QPL. Examine the contents of checkpoint directories and their significance for Delta streams. Gain insights into the marriage of Delta Lake and streaming, and discover why it's becoming increasingly popular among users building curated data lakes and end-to-end data pipelines.

Introduction
Sample Stream
Internals
Query Progress Log
Example Stream
Source Table
Source Table History
Query Progress
Streaming Checkpoint