Descending Through a Crowded Valley - Benchmarking Deep Learning Optimizers

Explore a comprehensive analysis of deep learning optimizers in this 41-minute video lecture. Dive into the complex world of optimization algorithms for neural networks, comparing 14 popular methods in a standardized benchmark. Learn about the challenges of selecting the right optimizer, the importance of hyperparameter tuning, and the impact of different algorithms on various deep learning tasks. Gain insights into learning rate schedules, noise effects, and practical recommendations for choosing optimizers. Understand the key findings of the study, including the variability of optimizer performance across tasks and the effectiveness of evaluating multiple optimizers with default parameters. Discover a reduced subset of competitive algorithms that can guide your future deep learning projects.

- Introduction & Overview
- The Overwhelming Amount of Optimizers
- Compared Optimizers
- Default Parameters & Tuning Distribution
- Deep Learning Problems Considered
- Tuning on Single Seeds
- Results & Interpretation
- Learning Rate Schedules & Noise
- Conclusions & Comments