YoVDO

Towards Understanding Modern Alchemy - Transformers as a Computational Model

Offered By: Simons Institute via YouTube

Tags

Transformers Courses Machine Learning Courses Computational Models Courses Finite Automata Courses Formal Languages Courses Language Models Courses In-context Learning Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the intricacies of in-context learning (ICL) in language models through a 37-minute lecture by Ekin Akyurek from MIT. Delve into the concept of in-context learning of formal languages (ICLL) as a model problem for understanding ICL. Examine how Transformers outperform recurrent and convolutional models in learning regular languages sampled from random finite automata. Discover the role of specialized "n-gram heads" in Transformers' superior performance and their potential to improve natural language modeling. Learn how incorporating these heads into neural models can enhance perplexity scores on datasets like SlimPajama. Gain insights into the computational capabilities of Transformers and their implications for advancing language model performance.

Syllabus

Towards Understanding Modern Alchemy


Taught by

Simons Institute

Related Courses

CMU Advanced NLP: How to Use Pre-Trained Models
Graham Neubig via YouTube
Stanford Seminar 2022 - Transformer Circuits, Induction Heads, In-Context Learning
Stanford University via YouTube
Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression
Simons Institute via YouTube
In-Context Learning: A Case Study of Simple Function Classes
Simons Institute via YouTube
AI Mastery: Ultimate Crash Course in Prompt Engineering for Large Language Models
Data Science Dojo via YouTube