Using Frameworks to Test Apache Airflow Data Pipelines

Discover how to effectively test Apache Airflow data pipelines using frameworks in this 46-minute conference talk from Conf42 Python 2024. Explore the fundamentals of Airflow, understand the importance of testing data pipelines, and learn about local development and testing techniques. Dive into Astro CLI and Airflow CLI testing features, including DAG validation, unit tests, and integration tests. Follow the journey of code from development to staging and production environments. Gain practical insights through a comprehensive demo and take away valuable lessons for implementing robust testing strategies in your Airflow projects.

intro
preamble
agenda
what is airflow? who is astronomer?
why test data pipelines?
local development and testing
astro cli and airflow cli testing features
dag.test
ci/cd and testing
from dev to staging
dag validation testing
unit tests with airflow
integration tests with airflow
the cf in ci/cd - our code on the way to the cloud
code getting promoted to prod
demo
take home message