Monitoring ML Models, IoT Data, and Quality Checks on Delta Lake - Quby's Approach

Explore a comprehensive 30-minute presentation on monitoring large-scale machine learning models, IoT streaming data, and automated quality checks using Delta Lake. Dive into Quby's innovative approach to managing Europe's largest energy dataset, consisting of petabytes of IoT data. Learn how Delta Lake ensures data quality through schema enforcement and evolution, and discover the crucial role of Data Engineers in verifying timely data ingestion with expected metrics. Examine the challenges of training and serving over half a million models daily, and understand the importance of balancing quality data with well-performing models. Gain insights into monitoring raw and processed data quality metrics using Databricks dashboards, tracking model performance with MLflow, and implementing Slack alerts for failure notifications. Explore real-world examples of managing large-scale data processing and machine learning pipelines in production environments.

Intro
What is Kiwi
Important facts
Waste checker service
Machine learning
Data curation
ML infrastructure
Data Monitoring
Importance of Monitoring
Spark Jobs
Silver Line
Monitoring Jobs
Monitoring Dashboard
Monitoring
Bronze Layer Table
Daily Data Injection
Job Configuration
Slack Integration
Data Quality Check
External Monitoring System
Summary
Outro