Building an Azure Data Analytics Platform End-to-End

Explore the comprehensive process of building a Microsoft Azure Data Analytics platform from scratch in this 48-minute conference talk by Paul Andrew, a Microsoft Data Platform MVP. Gain insights into the extensive knowledge required to create a complete Azure-based analytics solution, drawing from real-world experience. Learn about various aspects of data extraction, ingestion, transformation, and consumption across multiple Azure services. Discover how to scale your solution, choose appropriate compute types, and design an effective overall architecture. Delve into topics such as Synapse Analytics, Azure SQL Database, Stream Analytics, Spark, Event Hub, Data Lake, and Data Bricks. Understand key considerations for successful delivery, big data analytics, and data loading patterns and techniques in this informative session from SQLBits.

Intro
Question: What is our goal?
Paul's Reference Architecture
What is your primary design focus?
Data Extraction & Ingestion-Spec v1
Data Extraction & Ingestion - Solution 1
Data Extraction & Ingestion - Spec v2
Data Extraction & Ingestion - Solution 2
Data Extraction & Ingestion - Spec v3
Data Extraction & Ingestion - Solution 3
Data Extraction & Ingestion - Spec v4
Data Extraction & Ingestion - Solution 4
Data Extraction & Ingestion - Spec v5
Data Extraction & Ingestion - Solution 5
Data Extraction & Ingestion - Spec v6
Data Extraction & Ingestion - Solution 6
Scaling Up and/or Scaling Out
What Compute Type of Compute?
Data Transformation - Compute
Data Transformation - Storage & Format
Loading & Consuming Data
Consuming Our Lake House in Azure
Overall Architecture