Lessons Learned from Scaling Large Language Models in Production

Explore the challenges and solutions for scaling large language models (LLMs) in production environments during this 40-minute conference talk from MLOps World: Machine Learning in Production. Gain insights from Matt Squire, CTO of Fuzzy Labs, as he shares valuable lessons learned from building and running LLMs at scale for customers. Discover how to overcome the complexities of high-traffic demands, slow LLM inference, and expensive GPU resources. Learn about performance profiling techniques, optimizing GPU utilization, and implementing effective guardrails through real code examples. Understand the nuances of scaling beyond basic RAG applications with open-source models like Mistral, and acquire practical knowledge to enhance your LLM deployment strategies for production-level performance and efficiency.

Lessons learned from scaling large language models in production