YoVDO

Transformers Are RNNs- Fast Autoregressive Transformers With Linear Attention

Offered By: Yannic Kilcher via YouTube

Tags

Recurrent Neural Networks (RNN) Courses Deep Learning Courses Attention Mechanisms Courses Transformer Models Courses

Course Description

Overview

Explore a comprehensive video explanation of the paper "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention." Delve into the reformulation of the attention mechanism using kernel functions, resulting in a linear formulation that reduces computational and memory requirements. Discover the surprising connection between autoregressive transformers and RNNs. Learn about softmax attention, quadratic complexity, generalized attention mechanisms, kernels, linear attention, and experimental results. Gain insights into the intuition behind linear attention and understand the caveats of the RNN connection. This 48-minute video by Yannic Kilcher breaks down complex concepts, making them accessible to those interested in AI, attention mechanisms, transformers, and deep learning.

Syllabus

- Intro & Overview
- Softmax Attention & Transformers
- Quadratic Complexity of Softmax Attention
- Generalized Attention Mechanism
- Kernels
- Linear Attention
- Experiments
- Intuition on Linear Attention
- Connecting Autoregressive Transformers and RNNs
- Caveats with the RNN connection
- More Results & Conclusion


Taught by

Yannic Kilcher

Related Courses

Simple Recurrent Neural Network with Keras
Coursera Project Network via Coursera
Deep Learning: Advanced Natural Language Processing and RNNs
Udemy
Recurrent Neural Networks (RNNs) for Language Modeling with Keras
DataCamp
Deep Learning: Recurrent Neural Networks in Python
Udemy
Basics of Deep Learning
Udemy