Fine-tuning LLMs 30x Faster - Unsloth AI's Approach

Explore a comprehensive video interview with Daniel Han from Unsloth AI, delving into techniques for accelerating LLM fine-tuning by up to 30 times. Learn about Han's bug-hunting process, the use of Desmos for gradient checking, and an in-depth analysis of Gemma bugs. Discover insights on runtime bottlenecks, comparisons with llm.c, and discussions on multi-GPU support and reproducibility in machine learning research. Gain valuable knowledge on optimizing LLM performance and understanding the intricacies of fine-tuning processes in this informative hour-long session.

00:00:00 - Intro
00:01:40 - Hyperstack GPUs! sponsored
00:03:21 - About Daniel, getting started with ML
00:06:30 - Using Desmos to check grads
00:11:00 - Deep dive into Gemma bugs
00:38:00 - approximate GELU bug
00:50:00 - What are the bottlenecks to speeding up the runtime?
00:54:15 - comparison with llm.c?
00:58:30 - Is multi-GPU coming to Unsloth? :
01:00:00 - Reproducibility in ML research