Towards Understanding Modern Alchemy - Transformers as a Computational Model
Offered By: Simons Institute via YouTube
Course Description
Overview
Explore the intricacies of in-context learning (ICL) in language models through a 37-minute lecture by Ekin Akyurek from MIT. Delve into the concept of in-context learning of formal languages (ICLL) as a model problem for understanding ICL. Examine how Transformers outperform recurrent and convolutional models in learning regular languages sampled from random finite automata. Discover the role of specialized "n-gram heads" in Transformers' superior performance and their potential to improve natural language modeling. Learn how incorporating these heads into neural models can enhance perplexity scores on datasets like SlimPajama. Gain insights into the computational capabilities of Transformers and their implications for advancing language model performance.
Syllabus
Towards Understanding Modern Alchemy
Taught by
Simons Institute
Related Courses
Logic: Language and Information 1University of Melbourne via Coursera Logic: Language and Information 2
University of Melbourne via Coursera Language, Proof and Logic
Stanford University via edX 理论计算机科学基础 | Introduction to Theoretical Computer Science
Peking University via edX 离散数学概论 Discrete Mathematics Generality
Peking University via Coursera