Pioneering a Hybrid SSM Transformer Architecture - Jamba Foundation Model
Offered By: Databricks via YouTube
Course Description
Overview
Explore a groundbreaking conference talk on the development of Jamba, a novel Foundation Model based on a hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Delve into the decision-making process behind creating this innovative hybrid structure, and gain insights into its layered composition of SSM, Transformer, and MoE components. Learn how this flexible architecture enables resource- and objective-specific configurations, offering unprecedented throughput and the largest context window of 256K in its size class, while fitting 140K on a single GPU. Discover how Jamba introduces a paradigm shift in large language model development, presented by AI21 Labs CTO Barak Lenz. Access additional resources on LLM and MLOps, and connect with Databricks through various social media platforms for further exploration of cutting-edge AI technologies.
Syllabus
Pioneering a Hybrid SSM Transformer Architecture
Taught by
Databricks
Related Courses
GShard- Scaling Giant Models with Conditional Computation and Automatic ShardingYannic Kilcher via YouTube Learning Mixtures of Linear Regressions in Subexponential Time via Fourier Moments
Association for Computing Machinery (ACM) via YouTube Modules and Architectures
Alfredo Canziani via YouTube Stanford Seminar - Mixture of Experts Paradigm and the Switch Transformer
Stanford University via YouTube Decoding Mistral AI's Large Language Models - Building Blocks and Training Strategies
Databricks via YouTube