Running ML Inference Services in Shared Hosting Environments

Explore the challenges and solutions of running machine learning inference services in shared hosting environments like ECS and Kubernetes in this 26-minute conference talk from MLOps World. Learn how Nextdoor's ML team identified and resolved issues affecting latency and throughput, resulting in significant performance improvements. Discover key insights on request queue management and OpenMP parameter tuning to optimize ML inference services. Gain valuable knowledge from machine learning engineer Danny Luo's experience in implementing ML solutions in complex enterprise environments, and understand how to achieve substantial latency reductions, throughput increases, and improved resource utilization while maintaining performance in shared hosting setups.

Running ML Inference Services in Shared Hosting Environments