YoVDO

Training More Effective Learned Optimizers, and Using Them to Train Themselves - Paper Explained

Offered By: Yannic Kilcher via YouTube

Tags

Optimization Algorithms Courses Machine Learning Courses Deep Learning Courses Generalization Courses Implicit Regularization Courses

Course Description

Overview

Explore a comprehensive video analysis of a groundbreaking research paper on learned optimizers in machine learning. Delve into the concept of replacing traditional hand-crafted optimization algorithms with neural network-based learned optimizers capable of training a wide variety of problems without user-specified hyperparameters. Discover how these optimizers are trained on thousands of tasks, resulting in better generalization to unseen problems. Examine the unique behaviors exhibited by learned optimizers, including implicit regularization and adaptation to changing hyperparameters or architectures. Gain insights into the potential of these optimizers to train themselves from scratch and their implications for the future of machine learning optimization.

Syllabus

- Intro & Outline
- From Hand-Crafted to Learned Features
- Current Optimization Algorithm
- Learned Optimization
- Optimizer Architecture
- Optimizing the Optimizer using Evolution Strategies
- Task Dataset
- Main Results
- Implicit Regularization in the Learned Optimizer
- Generalization across Tasks
- Scaling Up
- The Learned Optimizer Trains Itself
- Pseudocode
- Broader Impact Statement
- Conclusion & Comments


Taught by

Yannic Kilcher

Related Courses

Understanding Deep Learning Requires Rethinking Generalization
University of Central Florida via YouTube
Implicit Regularization I
Simons Institute via YouTube
Benign Overfitting - Peter Bartlett, UC Berkeley
Alan Turing Institute via YouTube
Big Data Is Low Rank - Madeleine Udell, Cornell University
Alan Turing Institute via YouTube
Beyond Lazy Training for Over-parameterized Tensor Decomposition
Fields Institute via YouTube