YoVDO

nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training

Offered By: USENIX via YouTube

Tags

Deep Learning Courses Neural Networks Courses Parallel Computing Courses Model Training Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore a 14-minute conference talk from USENIX OSDI '24 that introduces nnScaler, a framework for generating efficient parallelization plans in deep learning training. Learn how nnScaler addresses limitations in existing search spaces by empowering domain experts to construct custom search spaces using three primitives: op-trans, op-assign, and op-order. Discover how this approach captures model transformation and temporal-spatial scheduling for any parallelization plan. Understand the application of constraints to prevent space explosion and how nnScaler can compose both existing and new search spaces. Examine experimental results demonstrating up to 3.5× speedup compared to solutions like DeepSpeed, Megatron-LM, and Alpa for popular DNN models such as SwinTransformer and AlphaFold2.

Syllabus

OSDI '24 - nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training


Taught by

USENIX

Related Courses

Neural Networks for Machine Learning
University of Toronto via Coursera Good Brain, Bad Brain: Basics
University of Birmingham via FutureLearn Statistical Learning with R
Stanford University via edX Machine Learning 1—Supervised Learning
Brown University via Udacity Fundamentals of Neuroscience, Part 2: Neurons and Networks
Harvard University via edX