Scaling Kubernetes: Best Practices for Managing Large-Scale Batch Jobs with Spark and Argo Workflow
Offered By: CNCF [Cloud Native Computing Foundation] via YouTube
Course Description
Overview
Discover best practices for managing large-scale batch jobs on Kubernetes with Spark and Argo Workflow in this informative conference talk. Learn how to build a Kubernetes cluster capable of frequently creating and deleting 20,000 pods for parallel computation. Explore techniques for optimizing the Kubernetes control plane, including list-watch mechanism, service broadcasting, environment variable attachments, and API server configurations. Gain insights into configuring Spark operator and Argo workflows controller for improved performance. Perfect for those managing data processing with Spark applications or genomics computing with Argo workflows, this presentation offers valuable guidance on scaling Kubernetes to handle significant workloads efficiently.
Syllabus
Scaling Kubernetes: Best Practices for Managing Large-Scale Batch Jobs... - Yu Zhuang & Liu Jiaxu
Taught by
CNCF [Cloud Native Computing Foundation]
Related Courses
Argo Workflows on Kubernetes - Core ConceptsUdemy Managing ML Models at Scale - Intuit’s ML Platform
USENIX via YouTube Automating Cloud-native Spark Jobs with Argo Workflows
Linux Foundation via YouTube Bringing ML Workflows to Heterogeneous Cloud Native Machine Learning Platforms Using Intermediate Representation
Linux Foundation via YouTube Multi-branch Pipeline with Argo Workflows and CI/CD Debugging
CNCF [Cloud Native Computing Foundation] via YouTube