Building a Data Science Platform with Argo Workflows

Explore how to automate thousands of data pipelines for multiple data teams with diverse requirements using Argo Workflows and a multi-cluster Kubernetes architecture in this 27-minute conference talk. Discover the unique approaches taken by Bloomberg and Pipekit to solve data operations at scale. Learn how to create reliable and scalable data science platforms that empower data scientists to self-serve their data engineering needs, from creating DAGs to analyzing logs and metrics. Gain insights into building efficient data pipelines that can deploy to multiple clusters across various regions, addressing the challenges of large-scale data operations in modern enterprises.

Building a Data Science Platform with Argo Workflows - J.P. Zivalich, Pipekit & Will Wang, Bloomberg