Rethinking Attention with Performers

Explore a comprehensive video explanation of the research paper "Rethinking Attention with Performers" in this 55-minute lecture. Dive deep into the innovative approach of using random positive orthogonal features to construct an unbiased estimator for the Attention matrix, achieving linear time complexity. Learn about the quadratic bottleneck in attention mechanisms, decomposition of the Attention matrix, and approximation of the softmax kernel. Discover why naive approaches fail and how positive and orthogonal features provide superior approximations. Examine experimental results, broader impacts, and implementation details including causal attention via prefix sums and code examples. Gain insights into the next generation of deep learning architectures that overcome memory and compute limitations of traditional Transformers.

- Intro & Outline
- Quadratic Bottleneck in Attention Mechanisms
- Decomposing the Attention Matrix
- Approximating the Softmax Kernel
- Different Choices, Different Kernels
- Why the Naive Approach does not work!
- Better Approximation via Positive Features
- Positive Features are Infinitely Better
- Orthogonal Features are Even Better
- Experiments
- Broader Impact Statement
- Causal Attention via Prefix Sums
- Code
- Final Remarks & Conclusion