Volcano: Introduction and Deep Dive into High-Performance Workload Management on Kubernetes

Explore a comprehensive introduction and deep dive into Volcano, a system for running high-performance workloads on Kubernetes, in this 32-minute conference talk by Klaus Ma from Huawei Cloud. Discover how Volcano addresses the challenges of batch scheduling in Kubernetes, providing powerful capabilities for ML/DL, big data applications, and Bioinformatics/Genomics. Learn about key concepts such as job management, resource management with queues, and dynamic resource sharing. Delve into various scheduling scenarios, including elastic scheduling, topology awareness, SLA scheduling, and CPU topology awareness. Gain insights into Volcano's application in batch scheduling for Spark, co-location and oversubscription, and its ability to handle large-scale clusters. Examine real-world use cases from Xiaohongshu's AI platform and Ruitian Investment's batch processing. Understand the project's community involvement and release cycle, equipping yourself with valuable knowledge for running high-performance workloads in Kubernetes environments.

Intro
Cloud Native for Intelligent Workload
Batch on K8s: Challenges
Volcano Overview
Key Concept
Job management
Resource management-Queue
Dynamic resource sharing between queues
Fair share within Queue
Case: hierarchical queue
Scheduling policy in Volcano
Scenario: Elastic scheduling
Scenario: Topology awareness
Scenario: SLA scheduling
Scenario: CPU Topology awareness
Scenario: batch scheduler for Spark
Scenario: Co-location and oversubscription
Scenario: Big scale cluster: 10k nodes, 1 million Pod
Use Case: Al platform at Xiaohongshu
Use Case: Batch at Ruitian investment
Community
Release