End-to-End Data Engineering Project

Learn how to create an end-to-end data engineering project using open tools from the modern data stack to turn scattered data into a model that drives insights and decision-making.

Introduction

• Transform complex data into insights
• What you should know

1. Project Overview and Preparation

• Project architecture overview
• Project setup
• Understanding the Big Star Collectibles database
• Setting up your data warehouse

2. Data Extraction and Loading

• Getting started with ELT tools: An introduction to Airbyte
• Deploying Airbyte for data synchronization
• Setting up sources and destinations in Airbyte
• Establishing connections in Airbyte
• Synchronizing and navigating through data

3. Starting Data Transformation and Modeling

• Introduction to data modeling with dbt
• Understanding the structure of a dbt project
• Initiating your dbt project
• Configuring data sources in dbt
• Challenge: Add a freshness check
• Solution: Add a freshness check

4. Data Transformation and Modeling

• Creating and customizing your dbt models
• Reviewing and executing dbt
• Securing your data with dbt tests
• Challenge: Add tests to the Marts model
• Solution: Add tests to the Marts model
• Automating documentation in dbt
• Completing your dbt project: A full development cycle

5. Data Orchestration

• Introduction to data orchestration with Dagster
• Integrating dbt models with Dagster assets
• Integrating Airbyte connections with Dagster assets
• Materializing assets using Dagit
• Challenge: Add a schedule to your data pipeline
• Solution: Add a schedule to your data pipeline

Conclusion

• An evolving field