Orchestrating Data Assets Instead of Tasks, With Dagster - Sandy Ryza
Offered By: Open Data Science via YouTube
Course Description
Overview
Discover the power of data orchestration in this insightful talk by Sandy Ryza, lead of the Dagster project at Elementl. Learn how orchestrators serve as the backbone for keeping data assets up-to-date and synchronized, from datasets to ML models. Explore the concept of data pipelines, delve into Apache Airflow, and understand the process of building and deploying pipelines. Gain valuable insights into the development lifecycle, including local development, unit testing, review and staging, as well as debugging and monitoring techniques. Perfect for data engineers, machine learning enthusiasts, and professionals interested in optimizing data synchronization and advanced analytics.
Syllabus
- Introductions
- What is a data pipeline?
- Apache Airflow
- Building a pipeline
- The development lifecycle
- Local development
- Unit/regression testing
- Review and staging
- Deploying
- Debugging/monitoring
- To sum up
- Q&A
Taught by
Open Data Science
Related Courses
Introduction, Principles and Origin of DagsterDatabricks via YouTube End-to-End Data Engineering Project
LinkedIn Learning Developing Data Pipelines with Branch Deployments - A New Approach
Databricks via YouTube Building Samsara's Machine Learning Platform with Ray
Anyscale via YouTube Rising Tides with Radical Transparency - Open Sourcing Your Data Platform
Data Council via YouTube