YoVDO

Swing: Short-cutting Rings for Higher Bandwidth Allreduce

Offered By: Scalable Parallel Computing Lab, SPCL @ ETH Zurich via YouTube

Tags

Distributed Systems Courses Machine Learning Courses Network Topologies Courses Supercomputers Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore a groundbreaking algorithm for improving allreduce performance on torus networks in this 26-minute conference talk from NSDI 2024. Delve into the innovative Swing algorithm, presented by Daniele De Sensi from the Scalable Parallel Computing Lab at ETH Zurich. Learn how Swing enhances allreduce bandwidth by maintaining low distances between communicating nodes through swinging between torus directions. Discover the algorithm's impressive performance gains, outperforming existing methods by up to 3x for various vector sizes and torus topologies. Gain insights into the potential impact of this research on machine learning workloads and high-performance computing systems, including Google TPUs, Amazon Trainium devices, and Top500 supercomputers.

Syllabus

Swing: Short-cutting Rings for Higher Bandwidth Allreduce


Taught by

Scalable Parallel Computing Lab, SPCL @ ETH Zurich

Related Courses

Tech Recruiting Foundations: 2 Understanding Hardware and Operating Systems
LinkedIn Learning
Linux Terminal for beginners
Udemy
How Do I Compute the Body Electric?
TED via YouTube
Analog Supercomputers - From Quantum Atom to Living Body
TEDx via YouTube
Self-Driving from Vision Only - Tesla's Self-Driving Progress by Andrej Karpathy - Talk Analysis
Yannic Kilcher via YouTube