Rapture of the Deep: Highs and Lows of Sparsity in Neural Networks

Explore the depths of sparsity in neural networks through this 37-minute conference talk by Remi Gribonval from INRIA, hosted by the Institut des Hautes Etudes Scientifiques (IHES). Delve into the natural promotion of sparse connections in neural networks for complexity control and potential interpretability guarantees. Compare classical sparse regularization for inverse problems with multilayer sparse approximation. Discover the role of rescaling-invariances in deep parameterizations, their advantages and challenges. Learn about life beyond gradient descent, including an algorithm that significantly speeds up learning of certain fast transforms via multilayer sparse factorization. Cover topics such as bilinear sparsity, blind deconvolution, ReLU network training with weight decay, optimization with support constraints, butterfly factorization, and the consequences of scale-invariance in neural networks.

Intro
Based on joint work with
Sparsity & frugality
Sparsity & interpretability
Deep sparsity?
Bilinear sparsity: blind deconvolution
ReLU network training - weight decay
Behind the scene
Greed is good?
Optimization with support constraints
Application: butterfly factorization
Wandering in equivalence classes
Other consequences of scale-invariance
Conservation laws