Seamless Flow: Evolving From Batch to Streaming Data Flows Using DLT

Explore how 84.51, a retail insights company, leverages Delta Live Tables (DLT) to build clickstream and customer order data flows in this 27-minute conference talk. Learn about the evolution from batch files to streaming data using DLT's Append Flows API, and discover techniques for combining streaming data sources with static historical data. Gain insights into using the Once Flow option for efficient duplicate removal between flows, and understand how to implement Change Data Capture and the sequence_by option to capture events in a specific order. Presented by Alli Hanlon and Scott Gordon, Data Engineers at 84.51˚, this talk demonstrates how DLT's declarative code and robust functionality can enhance data flow management in a production lakehouse environment.

Seamless Flow: Evolving From Batch to Streaming Data Flows Using DLT