Fixing GAN Optimization Through Competitive Gradient Descent - Anima Anandkumar

Explore the intricacies of GAN optimization through competitive gradient descent in this 35-minute lecture by Anima Anandkumar from Caltech. Delve into topics such as single agent optimization, competitive optimization, strategic equilibria, and applications in machine learning. Examine the concept of alternating gradient descent and learn how to linearize a game. Understand the principles of competitive gradient descent, compare it to existing methods, and investigate the solution of a GAN. Analyze different models of competing agents, including global, myopic, and predictive games. Conclude with a review of numerical results in this comprehensive talk from the Workshop on Theory of Deep Learning at the Institute for Advanced Study.

Intro
Single Agent Optimization
Competitive Optimization
Strategic Equilibria
Applications in ML
Alternating Gradient Descent
A polemic
Recall Gradient Descent
How to linearize a game?
Linear or Multilinear?
Competitive Gradient Descent
Why bilinear makes sense
What I think that they think that I think...
Comparison to existing methods
What is the solution of a GAN
Modeling competing agents
The global game
The myopic game
The predictive game
Numerical results