Regularization of Big Neural Networks

Explore a comprehensive guest lecture on regularization techniques for large neural networks delivered by Dr. Rob Fergus at the University of Central Florida. Delve into topics such as big neural nets, over-fitting, dropout and dropconnect methods, stochastic pooling, and deconvolutional networks. Learn about the theoretical analysis of these techniques, their limitations, and practical applications through experiments on datasets like MNIST, Street View House Numbers, and Caltech 101. Gain insights into convergence rates, the effects of network size and training set variations, and the link between deconvolutional networks and parts-and-structure models. Enhance your understanding of advanced deep learning concepts and their impact on computer vision tasks.

Intro
Big Neural Nets
Big Models Over-Fitting
Training with DropOut
DropOut/Connect Intuition
Theoretical Analysis of DropConnect
MNIST Results
Varying Size of Network
Varying Fraction Dropped
Comparison of Convergence Rates
Limitations of DropOut/Connect
Stochastic Pooling
Methods for Test Time
Varying Size of Training Set
Convergence / Over-Fitting
Street View House Numbers
Deconvolutional Networks
Recap: Sparse Coding (Patch-based)
Reversible Max Pooling
Single Layer Cost Function
Single Layer Inference
Effect of Sparsity
Effect of Pooling Variables
Talk Overview
Stacking the Layers
Two Layer Example
Link to Parts and Structure Models
Caltech 101 Experiments
Layer 2 Filters
Classification Results: Caltech 101
Deconvolutional + Convolutional
Summary