Fine-Tuning Giant Neural Networks on Commodity Hardware with Automatic Pipeline Model Parallelism
Offered By: USENIX via YouTube
Course Description
Overview
Explore a groundbreaking approach to fine-tuning giant neural networks on commodity hardware in this 14-minute conference talk from USENIX ATC '21. Delve into FTPipe, an innovative system that introduces a new dimension of pipeline model parallelism, making multi-GPU execution of fine-tuning tasks for massive neural networks accessible on standard equipment. Learn about the novel Mixed-pipe approach to model partitioning and task allocation, which allows for more flexible and efficient use of GPU resources without compromising accuracy. Discover how this technique achieves up to 3× speedup and state-of-the-art accuracy when fine-tuning giant transformers with billions of parameters, such as BERT-340M, GPT2-1.5B, and T5-3B, on commodity RTX2080-Ti GPUs. Gain insights into the potential of this technology to democratize access to state-of-the-art models pre-trained on high-end supercomputing systems.
Syllabus
USENIX ATC '21 - Fine-tuning giant neural networks on commodity hardware with automatic pipeline...
Taught by
USENIX
Related Courses
Natural Language ProcessingColumbia University via Coursera Natural Language Processing
Stanford University via Coursera Introduction to Natural Language Processing
University of Michigan via Coursera moocTLH: Nuevos retos en las tecnologías del lenguaje humano
Universidad de Alicante via Miríadax Natural Language Processing
Indian Institute of Technology, Kharagpur via Swayam