Scaling to Thousands of Data and CI/CD Pipelines Using Argo and Virtual Clusters

Explore how Pipekit provisions thousands of Kubernetes clusters for customers to submit Argo Workflows within minutes of registration in this 33-minute conference talk. Dive into the challenges and solutions developed through a partnership between Pipekit, co-maintainers of Argo Workflows, and Loft Labs, maintainers of vcluster, to create a fully-managed experience for users running data workloads at scale. Gain insights into architectural challenges, design choices, missteps, and scaling limits encountered while building Pipekit's Argo Workflows offering. Learn about secure tenant isolation on shared Kubernetes clusters, on-demand and automated cluster provisioning, optimal configurations for Argo Workflows and Argo CD with vcluster, and enabling useful metrics and observability features for each tenant's virtual cluster.

Scaling to Thousands of Data & CI/CD Pipelines Using Argo and Virtual... Tim Collins & Lukas Gentele