Create Complex DAGs and Task Dependencies with Apache Airflow

Learn to build complex data pipelines with Apache Airflow. This course will teach you to create DAGs that read, validate, aggregate, and load data while managing task dependencies, using XComs for passing data between tasks.

Apache Airflow excels in making it easy for you to manage complex data pipelines. In this course, Create Complex DAGs and Task Dependencies with Apache Airflow, you’ll gain the ability to design and implement intricate workflows in Apache Airflow. First, you’ll explore how to create a DAG that reads a CSV file from a local directory using a BashOperator to check if the file exists. Next, you’ll discover how to perform data validation and aggregation using PythonOperators. Finally, you’ll learn how to load the transformed data using a SQLiteOperator and set up task dependencies with the bitshift operator and set_upstream()/set_downstream() methods, controlling the flow of execution by passing data between tasks using XComs. When you’re finished with this course, you’ll have the skills and knowledge of Apache Airflow needed to create and manage complex DAGs and task dependencies efficiently.

• Course Overview 1min
• Managing Task Dependencies in a DAG 21mins