Demystifying Mixtral of Experts - Stanford CS25 Lecture

Explore the intricacies of Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model, in this illuminating lecture from Stanford's CS25 course. Delve into the architectural details of Mixtral, which builds upon the Mistral 7B framework but incorporates 8 feedforward blocks (experts) in each layer. Discover how the model's router network selects and combines outputs from two experts per token at each layer, allowing access to 47B parameters while actively using only 13B during inference. Gain insights into the expert routing decisions and their implications. Presented by Albert Jiang, an AI scientist at Mistral AI and PhD student at Cambridge University, this talk offers a deep dive into cutting-edge language model architecture and its applications in pretraining and reasoning.

Stanford CS25: V4 I Demystifying Mixtral of Experts