YoVDO

The Parallelism Tradeoff: Understanding Transformer Expressivity Through Circuit Complexity

Offered By: Simons Institute via YouTube

Tags

Transformers Courses Neural Networks Courses Computational Models Courses Complexity Theory Courses Circuit Complexity Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the computational power of transformer neural networks in this 45-minute lecture by Will Merrill from New York University. Delve into the proof that transformers with logarithmic arithmetic precision can be simulated by constant-depth logspace-uniform threshold circuits. Examine the implications of this finding on the limitations of transformers, such as their inability to solve certain problems if L≠P. Investigate the concept of a fundamental parallelism tradeoff, which suggests that highly parallelizable model architectures like transformers may have inherent weaknesses. Consider the potential implications of this tradeoff on the scaling paradigm in machine learning and natural language processing.

Syllabus

The Parallelism Tradeoff: Understanding Transformer Expressivity Through Circuit Complexity


Taught by

Simons Institute

Related Courses

Computational Complexity Theory
Indian Institute of Technology Kanpur via Swayam
Computational Complexity
IIT Hyderabad via Swayam
Proof and Circuit Complexity - Robert Robere
Institute for Advanced Study via YouTube
Quantum Complexity - Quantum Computation at CMU
Ryan O'Donnell via YouTube
Strong Average-Case Circuit Lower Bounds from Non-trivial Derandomization
Association for Computing Machinery (ACM) via YouTube