Creating Extensible Workflows with Off-Label Use of Python

Explore the intricacies of creating extensible workflows using unconventional Python techniques in this PyCon US talk by Benjy Weinberger. Dive into the challenges of designing a programmatic interface for workflow systems, balancing flexibility and constraints. Learn how the Pants build system leverages Python's type annotations, coroutines, and dataclasses to create an elegant extensibility API. Discover how these features can be combined to support complex functionalities like caching, concurrency, and remote execution without burdening task authors. Gain insights into workflow system design, task API challenges, and practical applications in data processing, ETL, and CI/CD. Follow along with real-world examples and understand how to apply these concepts to build more efficient and maintainable workflow-oriented systems.

Intro
About me
What is a workflow?
Workflows show up all over the place
Example: Processing Uploaded Images
Workflow is defined by a task graph
Workflow system design
Workflow for software builds
Design goals
Task API design challenges
Python to the rescue!
Building the rule graph
Static validation
Type annotations provide
Why coroutines?
Coroutines are powerful
Coroutines can express concurrency
Coroutines help avoid side effects
Coroutines provide natural control points for
Summary