Deep Neural Networks via Monotone Operators

Explore deep equilibrium models (DEQs) and monotone equilibrium models (monDEQs) in this 46-minute lecture by Zico Kolter for the International Mathematical Union. Delve into the monDEQ framework, which guarantees fixed point uniqueness and enables efficient operator splitting methods. Learn how to bound Lipschitz constants of monDEQ models, produce generalization bounds for these "infinitely deep" networks, and characterize their robustness. Discover the connections between monDEQs and mean-field inference, and understand how these approaches can formulate Boltzmann-machine-like models with guaranteed convergence to globally optimal solutions. Examine practical applications in language modeling and image segmentation, and gain insights into the theoretical and algorithmic challenges of DEQs.

Intro
The deep learning revolution recent examp
Deep Learning The story we all tell: deep learning algorithms build hierarchical models of input date, where the earlier layers create simple features and layer layers create high- level abstractions of the data
This talk
Outline
From deep networks to DEQs
Long history of related work
Implementing DEQS
The DEQ forward pass
How to train your DEQ
How to train your DED Compute gradients analytically via implicit function theorem
More information on implicit layers
Language modeling: WikiText-103
Multiscale deep equilibrium models
ImageNet Top-1 Accuracy
Citiscapes mlou
Visualization of Segmentation
Theoretical/algorithmic challenges for DE
Key result
Proof sketch for simpler case
Monotone operator equilibrium network
Initial study: CIFAR10
Additional points on monotone DEOS
Final thoughts