The State and Future of Cloud-Native Model Serving

Explore the state and future of cloud-native model serving in this 39-minute conference talk by Dan Sun from Bloomberg and Theofilos Papapanagiotou from Amazon. Gain insights into KServe, a cloud-native open-source project for serving production ML models built on CNCF projects like Knative and Istio. Learn about KServe's progress towards version 1.0, recent developments such as ModelMesh and InferenceGraph, and its future roadmap. Discover the Kubernetes design patterns used in KServe to achieve core ML inference capability, and understand its design philosophy and integration with the CNCF ecosystem. Examine how the InferenceService interface simplifies networking, lifecycle, and server configurations, enabling easy addition of serverless capabilities to model servers like TensorFlow Serving, TorchServe, and Triton on CPU/GPU. Explore scenarios demonstrating quick KServe implementation and evolution to production-ready setups with scalability, security, observability, and auto-scaling acceleration using CNCF projects such as Knative, Istio, SPIFFE/SPIRE, OpenTelemetry, and Fluid.

The State and Future of Cloud-Native Model Serving - Dan Sun, Bloomberg & Theofilos Papapanagiotou