YoVDO

Limitations of Attention Mechanism in Transformers - Implications for Generalization and Optimization

Offered By: Simons Institute via YouTube

Tags

Transformers Courses Machine Learning Courses Computational Models Courses Attention Mechanisms Courses Circuit Complexity Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the limitations of the attention mechanism in Transformer models through a 48-minute lecture by Bingbin Liu from Carnegie Mellon University. Delve into the study of Transformer's reasoning capabilities using sequential reasoning tasks formulated with finite-state automata. Discover how o(T)-layer Transformers can simulate T steps of sequential reasoning, utilizing tools from Krohn-Rhodes theory and circuit complexity. Examine empirical findings that reveal practical challenges in optimization and representation richness, preventing models from discovering optimal constructions. Investigate Transformers' struggles with out-of-distribution generalization on simple tasks easily solved by RNNs, highlighting two inherent limitations of the Transformer architecture. Gain insights into the implications of these limitations for generalization and optimization in Transformer models.

Syllabus

Limitations of attention mechanism, with implications in generalization and optimization


Taught by

Simons Institute

Related Courses

Introduction to Artificial Intelligence
Stanford University via Udacity
Natural Language Processing
Columbia University via Coursera
Probabilistic Graphical Models 1: Representation
Stanford University via Coursera
Computer Vision: The Fundamentals
University of California, Berkeley via Coursera
Learning from Data (Introductory Machine Learning course)
California Institute of Technology via Independent