Building a Lakehouse on the Microsoft Intelligent Data Platform

Explore the architecture and implementation of a Lakehouse on the Microsoft Intelligent Data Platform in this 45-minute SQLBits conference talk. Gain insights into spark-based engines and their utilization within Synapse Analytics. Delve into Delta, the underlying file format enabling the Lakehouse, and examine how Synapse compute engines interact with it. Learn about code automation, data transformation in Spark, orchestration patterns, and exposing Delta tables via serverless SQL. Discover considerations for Synapse serverless and Power BI model design. Presented by Microsoft Data Platform MVP Craig Porteous and Simon Whiteley, this session provides a comprehensive overview of building a modern data architecture using Azure technologies.

Intro
MICROSOFT INTELLIGENT DATA PLATFORM
THE COMPUTE ENGINES OF AZURE SYNAPSE
THE DATA LAKEHOUSE
THE SYNAPSE LAKEHOUSE
CODE AUTOMATION
TRANSFORMING DATA IN SPARK
GENERIC TRANSFORMATIONS
PARAMETER CELLS
APPLYING SPARK TO OUR ARCHITECTURE
ORCHESTRATION PATTERNS
EXPOSING DELTA TABLES VIA SERVERLESS SOL
SYNAPSE SERVERLESS CONSIDERATIONS
POWER BI MODEL DESIGN