Hessian-Aware Stochastic Differential Equation Modeling of SGD

Explore a 29-minute conference talk from the "One World Optimization Seminar in Vienna" workshop held at the Erwin Schrödinger International Institute for Mathematics and Physics (ESI) in June 2024. Delve into the development of a novel Stochastic Differential Equation (SDE) model for Stochastic Gradient Descent (SGD) that incorporates Hessian information. Learn about the Hessian-Aware Stochastic Modified Equation (HA-SME) and its advantages over existing SDE models in capturing SGD's escaping behaviors. Discover how HA-SME achieves improved approximation error guarantees and reduced dependence on smoothness parameters. Understand the significance of HA-SME as the first SDE model to exactly recover SGD dynamics for quadratic objectives under certain conditions.

Niao He - A Hessian-Aware Stochastic Differential Equation Modelling of SGD