How Fast Can Your Model Composition Run in Serverless Inference?

Explore the challenges and solutions for efficient multi-model composition and inference in serverless Kubernetes environments in this conference talk. Learn how the integration of BentoML with Dragonfly addresses slow deployment times, high operational costs, and scalability issues when serving interconnected suites of ML models. Discover a compelling case study of a RAG application combining LLM, embedding, and OCR models, showcasing efficient packaging and swift distribution through Dragonfly's innovative P2P network. Delve into the utilization of open-source technologies like JuiceFS and VLLM to achieve remarkable deployment times of just 40 seconds and establish a scalable blueprint for multi-model composition deployments. Gain insights into transforming the landscape of AI model serving and overcoming complexities in typical AI applications requiring multiple interconnected models.

How Fast Can Your Model Composition Run in Serverless Inference? - Fog Dong, BentoML & Wenbo Qi