YoVDO

Managing Millions of Tests Using Databricks - Automated Monitoring and Reporting System

Offered By: Databricks via YouTube

Tags

Databricks Courses Software Testing Courses CI/CD Courses Quality Assurance Courses Test Automation Courses Data Pipelines Courses Delta Lake Courses Bazel Courses

Course Description

Overview

Explore the challenges and solutions of managing millions of daily tests for Databricks Runtime in this 25-minute conference talk. Dive into the automated test monitoring and reporting system built using Databricks, learning how to ingest data from various sources like CI systems and Bazel build metadata into Delta. Discover techniques for analyzing test results, reporting failures to owners through Jira, and creating effective quality tracking reports. Gain insights into the deep technical stack, wide surface area, and guiding principles behind Databricks' testing approach. Learn about establishing test results and owners tables, building data pipelines, and implementing developer-friendly failure reporting. Understand how to connect problems with the right owners and use appropriate tools to solve complex testing challenges in large-scale data engineering and machine learning environments.

Syllabus

Intro
Deep technical stack
Wide surface area
Testing, testing, testing
Guiding principles
What is the actual problem?
Building data pipelines
Use the right tools for solving the problem
Establishing test results tables
Establishing test owners table
Reporting test failures to Jira
Test reporting pipeline
Connecting the problem with the right owner
Developer-friendly failure reporting


Taught by

Databricks

Related Courses

Data Processing with Azure
LearnQuest via Coursera
Mejores prácticas para el procesamiento de datos en Big Data
Coursera Project Network via Coursera
Data Science with Databricks for Data Analysts
Databricks via Coursera
Azure Data Engineer con Databricks y Azure Data Factory
Coursera Project Network via Coursera
Curso Completo de Spark con Databricks (Big Data)
Coursera Project Network via Coursera