Apache Airflow Essential Training

Explore how to work with the TaskFlow API, perform operations using TaskFlow, integrate PostgreSQL in Airflow, use sensors in Airflow, and work with hooks in Airflow.

Introduction

• Apache Airflow essential training

1. Working with the TaskFlow API

• Prerequisites
• Quick Airflow setup overview
• DAG using PythonOperators
• DAG using TaskFlow
• Passing data using XCom with operators
• Passing data using the TaskFlow API
• Tasks with multiple outputs
• Passing multiple outputs in TaskFlow

2. Performing Operations Using TaskFlow

• Branching using operators
• Branching using the TaskFlow API
• Complete branching pipeline using TaskFlow
• Interoperating and passing data between operators and TaskFlow
• Performing SQL operations with TaskFlow
• Data transformation and storage using TaskFlow

3. Integrating PostgreSQL in Airflow

• Installing PostgreSQL on macOS
• Installing PostgreSQL on WSL
• Connecting to PostgreSQL
• Using the PostgreSQL operator
• Performing PostgreSQL insert operations
• Performing PostgreSQL join operations
• A complete end-to-end pipeline with PostgreSQL
• Configuring PostgreSQL as a metadata database and using the LocalExecutor

4. Using Sensors in Airflow

• Introducing the FileSensor
• Creating a pipeline using the FileSensor
• Running DAGs with a FileSensor
• Creating and running a pipeline with SQLSensors

5. Working with Hooks in Airflow

• Setting up for a PostgreSQL pipeline with hooks
• Creating and running a pipeline with PostgresSQL hooks
• Setting up access to Amazon S3 buckets
• Setting up a connection to Amazon S3 buckets
• Creating and running a pipeline with an S3 hook

Conclusion

• Summary and next steps