Effortless Scalability: Orchestrating Large Language Model Inference with Kubernetes

Explore the intricacies of deploying and orchestrating large open-source inference models on Kubernetes in this 23-minute conference talk from CNCF. Discover how to automate the deployment of heavyweight models like Falcon and Llama 2 using Kubernetes Custom Resource Definitions (CRDs) for seamless management of large model files through container images. Learn about streamlining deployment with an HTTP server for inference calls, eliminating manual tuning of deployment parameters, and auto-provisioning GPU nodes based on specific model requirements. Gain insights into empowering users to deploy containerized models effortlessly by providing pod templates in the workspace custom resource inference field. Understand how the controller dynamically creates deployment workloads utilizing all GPU nodes, ensuring optimal resource utilization in the AI/ML landscape.

Effortless Scalability: Orchestrating Large Language Model Inference... Rohit Ghumare & Joinal Ahmed