Check-N-Run - A Checkpointing System for Training Deep Learning Recommendation Models

Explore a cutting-edge checkpointing system for training large-scale deep learning recommendation models in this NSDI '22 conference talk. Dive into the challenges of checkpointing massive ML models and discover how Check-N-Run addresses size and bandwidth issues. Learn about differential checkpointing techniques that track and save only modified parts of the model, particularly effective for recommendation models with embedding tables. Examine quantization strategies that significantly reduce checkpoint size without compromising training accuracy. Understand how these innovations lead to substantial reductions in required write bandwidth and storage capacity, improving checkpoint capabilities while lowering total ownership costs. Gain insights into the architecture of recommendation models, high-performance training at Meta, and the critical role of checkpointing in failure recovery and continuous learning for online training.

Intro
Recommendation Models are important . Use cases include
Recommendation Model Architecture
High Performance Training at Meta
The Criticality of Checkpointing • Failure recovery ensure progress
Checkpoint Challenges
Check-n-Run
Checkpointing Workflow
Reducing WB with Differential Checkpointing
Approaches for Differential Checkpointing • One-Shot Differential Checkpoint . Consecutive Incremental Checkpoint - Intermittent Differential Checkpoint
Checkpoint Quantization Compress checkpoint without degrading training accuracy
Comparing Quantization Strategies . Uniform quantization . Non-uniform quantization using kmeans • Adaptive uniform quantization
Quantization Bit-width Selection
Overall Reduction
Summary