Developing Elegant Workflows in Python Code with Apache Airflow

Learn to develop elegant workflows in Python using Apache Airflow in this EuroPython 2017 conference talk. Explore the concept of workflows as directed acyclic graphs (DAGs) and discover how Airflow enables you to bring block diagrams to life. Dive into basic Airflow concepts, including tasks, operators, and sensors. Through practical examples, master the art of defining workflows in Python code, implementing conditional execution, and leveraging Bash commands with templates. Gain insights into extending Airflow's functionality by creating custom task operators, sensors, and plugins. Understand how Airflow's resilient design handles task retries, workflow resumption, and log management through its user-friendly web interface.

Intro
ABOUT ME
WHAT IS A WORKFLOW?
A TYPICAL WORKFLOW
EXAMPLES EVERYWHERE
WORKFLOW MANAGERS
APACHE AIRFLOW
WHAT FLOWS IN A WORKFLOW?
SOURCE AND TRIBUTARIES
DISTRIBUTARIES AND DELTAS
BRANCHES
AIRFLOW CONCEPTS: DAGS
AIRFLOW CONCEPTS: OPERATOR
AIRFLOW CONCEPTS: SENSORS
AIRFLOW CONCEPTS: XCOM
SCAN FOR INFORMATION UPSTREAM
REUSABLE OPERATORS
CONDITIONAL EXECUTION
BASH COMMANDS AND TEMPLATES
AIRFLOW PLUGINS