Stephen Wright: Fundamentals of Optimization in Signal Processing

Explore the second lecture in a series on optimization fundamentals for signal processing, delivered by Stephen Wright. Delve into key topics including group sparsity, matrix inference problems, and various optimization algorithms. Learn about non-overlapping groups, tree-structured groups, and nuclear norm regularization. Examine steepest descent methods, backtracking, and convergence rates for different scenarios. Discover multistep methods like the Heavy-Ball algorithm and conjugate gradient, and gain insights into accelerated first-order methods. This comprehensive lecture provides essential knowledge for tackling complex optimization challenges in signal processing applications.

Group Sparsity
Three Scenarios
Non-overlapping Groups
Tree-Structured Groups
Matrix Inference Problems
Nuclear Norm Regularization
Another Matrix Inference Problem: Inverse Covariance
Atomic-Norm Regularization
References
Steepest Descent
Backtracking
Weakly convex: 1/k sublinear rate
Linear convergence without strong convexity
The slow linear rate is typical!
Multistep Methods: The Heavy-Ball
Conjugate Gradient
Accelerated First-Order Methods