Scaling ML Pipelines on Kubernetes Using Tekton - Overcoming Complexity

Explore the challenges and solutions for scaling machine learning pipelines on Kubernetes using Tekton in this 31-minute conference talk by Tommy Li from IBM. Delve into the limitations of Tekton Pipelines' native functionality for complex ML tasks and discover how custom tasks were developed in collaboration with the Tekton community to overcome single pod environment constraints. Learn about advanced ML scenarios, including data passing, network communication, and resource distribution, and how they can be handled more efficiently in a Kubernetes-native way. Gain insights into the integration of Tekton with KubeFlow Pipelines to eliminate storage bottlenecks and achieve truly scalable ML pipelines in Kubernetes environments.

The Complexity on Scaling ML Pipelines on Kubernetes Using Tekton - Tommy Li, IBM