Model Serving at the Edge - Challenges and Solutions with ModelMesh

Explore the challenges and solutions for deploying AI models on edge devices in this conference talk. Discover how ModelMesh, combined with K3s and MicroShift technologies, can simplify model serving at the edge. Learn about the multi-model serving backend of KServe and how it offers a small-footprint control-plane for managing model deployments on Kubernetes. Understand how ModelMesh utilizes multi-model runtimes with intelligent model loading/unloading to maximize limited resources while serving multiple models for inference. Gain insights into the generations of computing, machine learning lifecycle, complexities of model serving, and Kubernetes at the edge. Explore KServe's easy-to-use interfaces and standardized inference protocol. Dive into the ModelMesh architecture, serving runtimes, and its application on edge devices. Examine an example deployment and learn about higher density deployment challenges.

Intro
Outline
Generations of Computing
Machine Learning Lifecycle
Complexities of Model Serving
Kubernetes at the Edge
Introducing KServe
Easy to Use Interfaces
Kserve Standardized Inference Protocol
Enter ModelMesh
ModelMesh Architecture
ModelMesh Serving Runtimes
ModelMesh On Edge?
Example Deployment
Higher Density Deployment
Challenges