YoVDO

Lessons Learned from DAG-Based Workflow Orchestration

Offered By: MLOps World: Machine Learning in Production via YouTube

Tags

Data Pipelines Courses Python Courses MLOps Courses Data Engineering Courses Batch Processing Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the evolution of workflow orchestration in this insightful conference talk from MLOps World: Machine Learning in Production. Delve into the limitations of traditional Directed Acyclic Graph (DAG) based systems and discover how Prefect Orion (Prefect 2.0) addresses these constraints. Learn about the challenges data professionals face when fitting workflows into DAGs, including re-running partial workflows, executing long-running workflows, and dynamically adding tasks during runtime. Gain valuable insights from Kevin Gregory Kho, Senior Open Source Community Engineer at Prefect, as he discusses the transition to a DAG-less workflow orchestration system. Understand how Orion enhances developer experience by offering a more Pythonic interface and improved flexibility. Discover how this new approach allows for greater observability into specific tasks within workflows, ultimately leading to more efficient and adaptable data pipelines.

Syllabus

Lessons Learned from DAG based Workflow Orchestration


Taught by

MLOps World: Machine Learning in Production

Related Courses

Introduction to Windows PowerShell
Microsoft via edX
Windows PowerShell Basics
Microsoft via edX
Preparing for Google Cloud Certification: Cloud Data Engineer
Google Cloud via Coursera
Data Engineering on Google Cloud Platform en Français
Google Cloud via Coursera
Data Engineering on Google Cloud Platform en Español
Google Cloud via Coursera