Model Merging and Mixtures of Experts - AI in Production

Explore the cutting-edge techniques of model merging and Mixture of Experts (MoE) in this 11-minute conference talk from the AI in Production Conference. Dive into the popular open-source methods for combining fine-tuned models to create state-of-the-art LLMs. Learn the main concepts of model merging and gain hands-on experience implementing it using the mergekit library. Discover how to create your own models and upload them directly to the Hugging Face Hub with the provided notebook. Presented by Maxime Labonne, a Machine Learning Scientist at J.P. Morgan and Ph.D. holder from the Polytechnic Institute of Paris, this talk covers topics such as merging techniques, Slurp, DAT, Franken merges, and merging recipes. Gain valuable insights from an expert in the field and expand your knowledge of advanced LLM techniques.

Intro
Welcome
Why Merging
Merging Techniques
Slurp
DAT Ties
Path Through Technique
Franken merges
Mixture of experts
Merging recipes
Merging library
Fine Tuning
Conclusion
Outro