Insights on Gradient-Based Algorithms in High-Dimensional Learning

Explore gradient-based algorithms in high-dimensional learning through this Richard M. Karp Distinguished Lecture. Delve into the analysis of gradient descent algorithms and their noisy variants in nonconvex settings. Examine several high-dimensional statistical learning problems where gradient-based algorithm performance can be analyzed precisely. Discover how statistical physics provides exact closed solutions for algorithm performance in the high-dimensional limit. Cover topics including the spiked mixed matrix-tensor model, perceptron, and phase retrieval. Gain insights into dynamical mean-field theory, Langevin dynamics, and stochastic gradient descent. Investigate phase diagrams, landscape analysis, and the teacher-student perceptron model. Understand the behavior of gradient descent in phase retrieval and explore theories for over-parameterized landscapes.

Intro
WORKHORSE OF MACHINE LEARNIN
IN DEEP LEARNING
STRATEGY
WHY THIS MODEL?
ESTIMATORS
GRADIENT-BASED ALGORITHMS
DYNAMICAL MEAN FIELD THEORY
LANGEVIN STATE EVOLUTION (NUMERICAL SOLUTION)
LANGEVIN PHASE DIAGRAM
GRADIENT-FLOW PHASE DIAGRAM
POPULAR "EXPLANATION"
SPURIOUS MINIMA DO NOT NECESSARILY CAUSE GF TO FAIL
WHAT IS GOING ON?
TRANSITION RECIPE
TRANSITION CONJECTUR
LANDSCAPE ANALYSIS
CONCLUSION ON SPIKED MATRIX-TENSOR MODEL
TEACHER-NEURAL SETTING
TEACHER STUDENT PERCEPTRON
PHASE RETRIEVAL: OPTIMAL SOLUTION
GRADIENT DESCENT FOR PHASE RETRIEVAI
PERFORMANCE OF GRADIENT DESCENT
GRADIENT DESCENT NUMERICALLY
TOWARDS A THEORY
OVER-PARAMETRISED LANDSPACE
STOCHASTIC GRADIENT DESCENT
DYNAMICAL MEAN-FIELD THEOR Mignaco, Urbani, Krzakala, LZ, 2006.06098
DMFT FOLLOWS THE WHOLE TRAJECTORY