Matrix Calculus for Machine Learning and Beyond
Offered By: Massachusetts Institute of Technology via MIT OpenCourseWare
Course Description
Overview
We all know that calculus courses such as [*18.01 Single Variable Calculus*](https://ocw.mit.edu/courses/18-01sc-single-variable-calculus-fall-2010/) and [*18.02 Multivariable Calculus*](https://ocw.mit.edu/courses/18-02sc-multivariable-calculus-fall-2010/) cover univariate and vector calculus, respectively. Modern applications such as machine learning and large-scale optimization require the next big step, "matrix calculus" and calculus on arbitrary vector spaces.
This class covers a coherent approach to matrix calculus showing techniques that allow you to think of a matrix holistically (not just as an array of scalars), generalize and compute derivatives of important matrix factorizations and many other complicated-looking operations, and understand how differentiation formulas must be reimagined in large-scale computing.
Syllabus
- Lecture 1 Part 1: Introduction and Motivation
- Lecture 1 Part 2: Derivatives as Linear Operators
- Lecture 2 Part 1: Derivatives in Higher Dimensions: Jacobians and Matrix Functions
- Lecture 2 Part 2: Vectorization of Matrix Functions
- Lecture 3 Part 1: Kronecker Products and Jacobians
- Lecture 3 Part 2: Finite-Difference Approximations
- Lecture 4 Part 1: Gradients and Inner Products in Other Vector Spaces
- Lecture 4 Part 2: Nonlinear Root Finding, Optimization, and Adjoint Gradient Methods
- Lecture 5 Part 1: Derivative of Matrix Determinant and Inverse
- Lecture 5 Part 2: Forward Automatic Differentiation via Dual Numbers
- Lecture 5 Part 3: Differentiation on Computational Graphs
- Lecture 6 Part 1: Adjoint Differentiation of ODE Solutions
- Lecture 6 Part 2: Calculus of Variations and Gradients of Functionals
- Lecture 7 Part 1: Derivatives of Random Functions
- Lecture 7 Part 2: Second Derivatives, Bilinear Forms, and Hessian Matrices
- Lecture 8 Part 1: Derivatives of Eigenproblems
- Lecture 8 Part 2: Automatic Differentiation on Computational Graphs
Taught by
Prof. Alan Edelman and Prof. Steven G. Johnson
Tags
Related Courses
Introduction to Neural Networks and PyTorchIBM via Coursera Regression with Automatic Differentiation in TensorFlow
Coursera Project Network via Coursera Neural Network from Scratch in TensorFlow
Coursera Project Network via Coursera Customising your models with TensorFlow 2
Imperial College London via Coursera PyTorch Fundamentals
Microsoft via Microsoft Learn