Supercharging Pipeline Efficiency with ML Performance Prediction

Explore how Singular enhanced pipeline efficiency using machine learning for task performance prediction in this 25-minute PyCon US talk. Learn about the challenges of processing large-scale customer data with varying task durations and resource requirements. Discover how the implementation of a machine learning model to predict task duration and memory usage, combined with Celery's advanced routing and Kubernetes' autoscaling capabilities, led to improved pipeline performance and reduced cloud costs. Gain insights into overcoming infrastructure issues, minimizing interruptions, and optimizing resource allocation in a high-volume data processing environment.

Talks - Boaz Wiesner, Keren Meron: Supercharging Pipeline Efficiency with ML Performance Prediction