Generalization in Deep Learning Through the Lens of Implicit Rank Minimization

Explore a lecture on the implicit regularization in deep learning through the lens of rank minimization. Delve into the theoretical analysis of matrix and tensor factorizations, equivalent to certain linear and non-linear neural networks. Discover how gradient-based optimization tends to fit training data with predictors of low complexity, leading to generalization. Examine dynamical characterizations establishing implicit regularization towards low matrix and tensor ranks, challenging prior beliefs about norm minimization. Consider the implications of these findings for both theory and practice in modern deep learning, highlighting the potential of ranks to explain and improve generalization. Learn about matrix completion, tensor completion, and their connections to linear and non-linear neural networks. Investigate practical applications, including rank minimization in neural network layers and potential explanations for generalization on natural data.

Intro
Generalization via Bis-Variance Tradeoff
Generalization in Deep Learning
Linear Models: Implicit Norm Minimization Linear Regression
Implicit Norm Minimization In Deep Learning?
Perspective: Implicit Rank Minimization
Outline
Matrix Completion Two-Dimensional Prediction
MF Linear NN
Conjecture: Implicit Nuclear Norm Minimization
Dynamical Analysis of Implicit Regularization in MF
Implicit Regularization in MF Norm Minimization Does the implicit regularization in MF minimize a norm?
Drawbacks of Studying MF
Tensor Completion Multi-Dimensional Prediction
TF Shallow Non-Linear Convolutional NN
Dynamical Analysis of Implicit Regularization in TF
Analogy Between Implicit Regularizations
HTF Deep Non-Linear CNN TF does not account for depth
Dynamical Analysis of Implicit Regularization in HTF
Practical Application: Rank Minimization in NN Layers
Potential Explanation for Generalization on Natural Data
Countering Locality of CNNs via Regularization
Recap
Implicit Rank Minimization in Deep Learning