Data Engineering Foundations

Learn the key facets of data engineering, from its place in the data science realm, to the specific tasks and skills every data engineer should possess.

Introduction

• What is data engineering?

1. Introduction to Data Engineering

• Introduction to data engineering
• Data engineer vs. data scientist
• Essential tools for data engineering

2. Databases and Dataframes

• Intro to databases and their types
• Understanding database schema
• Distributive computing

3. Data Engineering Tools

• MapReduce and Hadoop
• Hive
• Spark
• Airflow

4. ETL Pipelines

• Sources of data extraction
• Data extraction from a PostgreSQL database
• Challenge: Data extraction
• Solution: Data extraction
• Transforming data
• Challenge: Transforming data
• Solution: Transforming data
• Loading data into a DB
• Challenge: Loading data
• Solution: Loading data
• Scheduling ETL pipeline using Airflow

Conclusion

• Next steps