Scaling Python for Machine Learning - Beyond Data Parallelism

Explore advanced techniques for scaling Python in machine learning beyond data parallelism in this conference talk from GOTO Chicago 2023. Dive into the world of distributed computing with Holden Karau, an Open Source Engineer at Netflix, as she examines Spark, Dask, and Ray for scaling machine learning models. Learn about distributed tasks, actor models for managing model weights during training, and fault tolerance in various frameworks. Gain insights into the similarities and differences between Dask and Ray distributed tasks, understand task and actor fault tolerance, and discover the relationship between Ray and Netflix. This comprehensive presentation covers topics such as data parallelism refresher, distributed task structures, Spark's capabilities, and actor fault tolerance in Ray and Dask, providing valuable knowledge for scaling Python applications in machine learning contexts.

Intro
Probable relevant biases
Quick refresher on data parallelism
What do distributed tasks look like?
Dask distributed tasks
Ray distributed tasks
How are they different & same?
Task fault tolerance
Does Spark have tasks & actors?
Ray Diagram
Ray actor fault tolerance
What's up with Ray & Netflix?
Dask actor fault tolerance
Outro