Hands-On Introduction: Data Engineering

Discover and apply the fundamentals of data engineering in this hands-on, approachable, skills-based course.

Introduction

• Hands-on data engineering
• Background knowledge
• The history of data in the enterprise
• Using GitHub Codespaces with this course

1. Getting Your Bearings

• Data engineering and data pipelines
• Orchestration: In the modern sense
• Extract, transform, load (ETL)
• Tasks, DAGs, and dependencies

2. Airflow Introduction

• Introduction to Airflow
• Installing Airflow
• Running the Airflow webserver and scheduler
• Adjusting Airflow configuration settings
• Build a 1 Task DAG
• Build a 2 Task DAG

3. Basic ETL in Airflow

• ETL in Airflow
• A real-world data engineering task
• Extracting data manually
• Extracting data with Airflow
• Transforming data manually
• Transforming data with Airflow
• Loading data manually
• Loading data with Airflow
• Building an ETL DAG with Airflow
• Challenge: Review ETL questions
• Solution: Solutions to ETL questions

Conclusion

• The future of data engineering