Understanding Deep Learning Requires Rethinking Generalization

Explore the intricacies of deep learning and challenge conventional wisdom on generalization in this 40-minute lecture from the University of Central Florida. Delve into topics such as the Universal Approximation Theorem, L2 Regularization, Dropout, and Data Augmentation. Examine randomization tests and their results, leading to thought-provoking conclusions and implications. Investigate explicit and implicit regularization techniques, finite-sample expressivity of neural networks, and draw comparisons to linear models. Conclude by analyzing the role of Stochastic Gradient Descent (SGD) in deep learning, ultimately reshaping your understanding of generalization in neural networks.

Intro
Presentation Outline
Universal Approximation Theorem
L2 Regularization - "Weight Decay"
Dropout
Data Augmentation
Rondomization Tests
Results of Randomization Tests
Conclusions & Impications
Explicit Regularization Tests
Implicit Regularization Findings
Finite-Sample Expressivity of Neural Networks
Appeal to Linear Models
Investigating SGD
Final Conclusions