Movement Pruning - Adaptive Sparsity by Fine-Tuning

Explore an in-depth analysis of Movement Pruning, a novel approach to adaptive sparsity in deep neural networks. Delve into the limitations of traditional Magnitude Pruning in transfer learning scenarios and discover how Movement Pruning offers a superior solution. Learn about the mathematical foundations, experimental results, and potential improvements through distillation. Gain insights into the analysis of learned weights and understand how this method can significantly reduce model size while maintaining high performance, particularly in large pretrained language models. Compare various pruning techniques and their effectiveness in different machine learning contexts.

- Intro & High-Level Overview
- Magnitude Pruning
- Transfer Learning
- The Problem with Magnitude Pruning in Transfer Learning
- Movement Pruning
- Experiments
- Improvements via Distillation
- Analysis of the Learned Weights