GenAI Pipeline for Content Generation with Apache Airflow

Explore a comprehensive conference talk on building a GenAI pipeline for content generation using Apache Airflow. Dive into the challenges of creating GenAI pipelines and discover key Airflow features that address these issues. Learn about the TaskFlow API, automatic retries, branching, deferrable operators, dataset scheduling, and dynamic task mapping. Follow along with a detailed demo of a content generation repository and gain insights into Astro, a managed Airflow service. Walk away with practical knowledge on leveraging Apache Airflow for efficient GenAI pipeline development and management.

intro
preamble
agenda
airflow
challenges when creating genai pipelines
key airflow features for genai
taskflow api - airflow decorators
automatic retries in airflow
branching in airflow
deferrable operators
dataset scheduling
datasets in the airflow 2.9 ui
advanced dataset scheduling
dynamic task mapping
demo repo: content generation
astro
take home message