YoVDO

Self - Cross, Hard - Soft Attention and the Transformer

Offered By: Alfredo Canziani via YouTube

Tags

Transformer Architecture Courses Deep Learning Courses Neural Networks Courses PyTorch Courses Jupyter Notebooks Courses Self-Attention Mechanisms Courses

Course Description

Overview

Explore the intricacies of attention mechanisms and Transformer architecture in this comprehensive lecture. Delve into self-attention, cross-attention, hard attention, and soft attention concepts. Learn about set encoding use cases and the key-value store paradigm. Understand the implementation of queries, keys, and values in both self-attention and cross-attention contexts. Examine the Transformer's encoder-predictor-decoder architecture, with a focus on the encoder and the unique "decoder" module. Gain practical insights through a PyTorch implementation of a Transformer encoder using Jupyter Notebook. Additionally, discover useful tips for reading and summarizing research papers collaboratively.

Syllabus

– Welcome to class
– Listening to YouTube from the terminal
– Summarising papers with @Notion
– Reading papers collaboratively
– Attention! Self / cross, hard / soft
– Use cases: set encoding!
– Self-attention
– Key-value store
– Queries, keys, and values → self-attention
– Queries, keys, and values → cross-attention
– Implementation details
– The Transformer: an encoder-predictor-decoder architecture
– The Transformer encoder
– The Transformer “decoder” which is an encoder-predictor-decoder module
– Jupyter Notebook and PyTorch implementation of a Transformer encoder
– Goodbye :


Taught by

Alfredo Canziani

Tags

Related Courses

Axial-DeepLab - Stand-Alone Axial-Attention for Panoptic Segmentation
Yannic Kilcher via YouTube
Linformer - Self-Attention with Linear Complexity
Yannic Kilcher via YouTube
Synthesizer - Rethinking Self-Attention in Transformer Models
Yannic Kilcher via YouTube
The Narrated Transformer Language Model
Jay Alammar via YouTube
Learning the Structure of EHR with Graph Convolutional Transformer - Edward Choi
Stanford University via YouTube