Scalable Evaluation and Serving of Open Source LLMs

Explore the challenges and solutions for serving open source Large Language Models (LLMs) in production environments through this insightful conference talk. Delve into the progress made in developing open source LLMs and the current limitations in systems designed to deploy these models effectively. Learn from Dr. Waleed Kadous, engineering leader at Anyscale, as he shares his expertise on tackling the complexities of implementing LLMs in real-world applications. Gain valuable insights into system architecture, scalability considerations, and best practices for evaluating and serving open source LLMs. Discover how to leverage tools like Ray, an open-source project for building scalable AI and Python applications, to overcome production hurdles. Benefit from the speaker's extensive experience in machine learning, location services, and system architecture at companies like Anyscale, Uber, and Google.

Scalable Evaluation and Serving of Open Source LLMs // Waleed Kadous // LLMs in Prod Conference