Neural Networks as Interacting Particle Systems

Explore the intersection of machine learning, neural networks, and complex systems modeling in this 43-minute lecture by Eric Vanden-Eijnden at the Alan Turing Institute. Delve into the emerging paradigm of combining statistical inference, high-throughput computation, and physical laws to tackle complex models in science, engineering, and social sciences. Discover how neural networks can be viewed as interacting particle systems, and examine topics such as the 3-spin model on high-dimensional spheres, functional formulation for large networks, and error scaling using the Central Limit Theorem. Investigate the discrete training set approach, stochastic gradient descent, and the limiting stochastic differential equation. Learn about Dean's equation for particles with correlated noise and explore learning techniques with Gaussian kernels and single-layer networks with sigmoid nonlinearity. Gain insights into the mathematical foundations of data-driven modeling and its applications in collective dynamics, molecular modeling, cell biology, and fluid dynamics.

Intro
Machine learning and neural networks
3-spin model on the high-dimensional sphere
Neural networks and approximation theory
Functional formulation in the limit of large n
Parameters as particles with loss function as interacting potential
Error scaling - Central Limit Theorem (CLT)
Discrete training set and stochastic gradient descent
Limiting stochastic differential equation for SGD
Dean's equation for particles with correlated noise
Learning with Gaussian kemels
Learning with single layer networks with sigmoid nonlinearity