MLflow at Company Scale - Scaling and Optimizing for High-Volume Machine Learning

Discover how to implement MLflow at enterprise scale in this 27-minute talk from Databricks. Learn about managing 50,000 runs, millions of metrics, parameters, and tags, with bursts up to 20,000 QPS. Explore the setup of a shared MLflow instance at Criteo, including contributions to SQLAlchemyStore for improved responsiveness. Gain insights into transforming MLflow into a production-ready system, horizontal scaling on a Mesos cluster, and implementing Prometheus-based monitoring. Understand the integration of company Single Sign-On (SSO) authentication and how data scientists register runs from Europe's largest Hadoop cluster. Dive into topics such as architecture, scaling SQLAlchemyStore, whitelisting columns, configuring Gunicorn applications, automatic database migration, periodic jobs, and JavaScript JWT integration.

Intro
Machine Learning at Criteo
Architecture
Scale SOLAlchemyStore
Whitelist Column
Monitoring
Configure the Gunicorn Application
Automatic DB migration
Periodic Jobs
JWT Integration in Javascript
Integration to company SSO
Write Easily Fitler Queries
Miflow-yarn
Code Organization