Approximation Theory of Deep Learning from the Dynamical Viewpoint

Explore a 45-minute lecture on the approximation theory of deep learning from a dynamical perspective, delivered by Qianxiao Li from the National University of Singapore. Delve into the intersection of machine learning and dynamical systems, examining how composition in deep learning can be viewed as dynamics. Investigate supervised learning, the problem of approximation, and universal approximation by dynamics. Analyze sequence modeling applications, including deep learning architectures for sequence modeling and the approximation theory for sequence modeling. Examine recurrent neural network hypothesis spaces, their properties, and approximation guarantees. Gain insights into convolutional and encoder-decoder architectures, and discover how the analysis of recurrent neural networks can be extended to these structures.

Intro
Deep Learning: Theory vs Practice
Composition is Dynamics
Supervised Learning
The Problem of Approximation
Example: Approximation by Trigonometric Polynomials
The Continuum Idealization of Residual Networks
How do dynamics approximate functions?
Universal Approximation by Dynamics
Approximation of Symmetric Functions by Dynamical Hypothesis Spac
Sequence Modelling Applications
DL Architectures for Sequence Modelling
Modelling Static vs Dynamic Relationships
An Approximation Theory for Sequence Modelling
The Recurrent Neural Network Hypothesis Space
The Linear RNN Hypothesis Space
Properties of Linear RNN Hypothesis Space
Approximation Guarantee (Density)
Smoothness and Memory
Insights on the (Linear) RNN Hypothesis Space
Convolutional Architectures
Encoder-Decoder Architectures
Extending the RNN Analysis