Lessons Learned from DAG-Based Workflow Orchestration
Offered By: MLOps World: Machine Learning in Production via YouTube
Course Description
Overview
Explore the evolution of workflow orchestration in this insightful conference talk from MLOps World: Machine Learning in Production. Delve into the limitations of traditional Directed Acyclic Graph (DAG) based systems and discover how Prefect Orion (Prefect 2.0) addresses these constraints. Learn about the challenges data professionals face when fitting workflows into DAGs, including re-running partial workflows, executing long-running workflows, and dynamically adding tasks during runtime. Gain valuable insights from Kevin Gregory Kho, Senior Open Source Community Engineer at Prefect, as he discusses the transition to a DAG-less workflow orchestration system. Understand how Orion enhances developer experience by offering a more Pythonic interface and improved flexibility. Discover how this new approach allows for greater observability into specific tasks within workflows, ultimately leading to more efficient and adaptable data pipelines.
Syllabus
Lessons Learned from DAG based Workflow Orchestration
Taught by
MLOps World: Machine Learning in Production
Related Courses
Introduction to Windows PowerShellMicrosoft via edX Windows PowerShell Basics
Microsoft via edX Preparing for Google Cloud Certification: Cloud Data Engineer
Google Cloud via Coursera Data Engineering on Google Cloud Platform en Français
Google Cloud via Coursera Data Engineering on Google Cloud Platform en Español
Google Cloud via Coursera