DataDevOps for the Modern Data Warehouse on Microsoft Azure

Explore DevOps principles applied to Big Data Analytical Pipelines in this comprehensive conference talk. Dive into the Modern Data Warehouse architecture, designed to address the challenges of Big Data, Machine Learning, and Advanced Analytics. Learn how to build CI/CD pipelines for multi-source data warehouses using Microsoft Azure Data Platform technologies, including Data Factory, Databricks, Data Lake Gen2, and AzureDevOps. Through extensive demonstrations, gain insights into validating data, ensuring data quality, and implementing DevOps practices in data pipelines. Discover how to leverage Azure Key Vault, create release pipelines, manage environments, and utilize PowerShell scripts for efficient data operations. Explore monitoring techniques using Application Insights and gain a deeper understanding of the Modern Data Warehouse ecosystem.

Introduction
Agenda
Traditional Data Warehouse
Data Lake
Data Lake tiers
Validate
Ensure
Demo
Common Question
DevOps
Data Pipelines
Data Bricks
Sequel Data Warehouse
Data Factory
Data Factory Workflow
Key Vault
Sample Release
Commit to Master
Static artifacts
Seco packages
Creating a release
Release Pipeline
Staging Environment
Variables
Release
Override parameters
PowerShell script
Backpack task
Monitoring
Monitoring Metrics
Application Insights
Summary