From Classical Statistics to Modern ML - The Lessons of Deep Learning - Mikhail Belkin

Explore a thought-provoking lecture on the evolution of machine learning theory, from classical statistics to modern deep learning approaches. Delve into key concepts such as Empirical Risk Minimization, uniform laws of large numbers, and capacity control. Examine the intriguing U-shaped generalization curve and challenge conventional wisdom about overfitting in interpolation. Discover why deep learning interpolation has become best practice and investigate the "double descent" risk curve phenomenon. Analyze random Fourier networks, interpolated k-NN schemes, and the relationship between interpolation and adversarial examples. Gain insights into the landscape of generalization, optimization techniques, and the power of interpolation in modern machine learning frameworks. Learn valuable lessons from deep learning applications and explore the potential of fast and effective kernel machines.

Intro
Empirical Risk Minimization
The ERM/SRM theory of learning
Uniform laws of large numbers
Capacity control
U-shaped generalization curve
Does interpolation overfit?
Interpolation does not overfit even for very noisy data
why bounds fail
Interpolation is best practice for deep learning
Historical recognition
where we are now: the key lesson
Generalization theory for interpolation?
Interpolated k-NN schemes
Interpolation and adversarial examples
"Double descent" risk curve
Random Fourier networks
what is the mechanism?
Is infinite width optimal?
Smoothness by averaging
Double Descent in Random Feature settings
Framework for modern ML
The landscape of generalization
optimization: classical
The power of interpolation
Learning from deep learning: fast and effective kernel machines
Points and lessons