The Data Lakehouse for Streaming Data - A Talk for Everyone Who Loves Data

Explore the technical aspects of Data Lakehouses and their impact on streaming data in this comprehensive talk. Delve into the fusion of data lakes and data warehouses, examining how open-source projects like Delta Lake enhance data management with ACID transactions, schema enforcement, and efficient metadata handling. Discover the capabilities of open-source solutions for streaming data and gain insights into future improvements. Investigate streaming data analysis, machine learning on the lakehouse, and Project Lightspeed's potential for low-latency Apache Spark Structured Streaming. Witness a live demonstration of Twitter stream ingestion using a declarative, auto-scaling data pipeline for sentiment analysis with Hugging Face. Ideal for data architects, engineers, and practitioners interested in open-source and cloud services, this presentation offers a deep dive into the Databricks Lakehouse and its practical applications.

Intro
About Data Lakhouse
First wow moment
High performance computing
Streaming data
Stream processing
Technical advantages
Misconceptions
Cap Theorem
Summary
Spark Structured Streaming
Project Lightspeed
Streaming ETL
Data Lakhouse
Data Maturity Curve
Two Paradigms
The Lakhouse
How is this working
Google for Lakhouse
Delta Lake
Delta
Does it work
Short answer
Autoloader
Orchestration
Daytona I Summit
Battalion Architecture
The Next Step
Corona
I was sick
I built a simulator
DLT pipeline
Kafka
Faker
Streaming Pipeline
Data Pipeline
Data Analytics
Simple Asset Transaction
Data Warehouse