On-Demand Systems and Scaled Training Using the JobSet API

Explore the JobSet API for orchestrating complex workflows in ephemeral environments through this informative conference talk. Discover how to efficiently manage large-scale machine learning model training and build on-demand HPC systems using this powerful tool. Learn about automating the setup of training workloads with common frameworks like PyTorch and see results from large-scale experiments utilizing thousands of TPU chips. Gain insights into streamlining the process of creating on-demand HPC systems and establishing standardized environments for experimental comparisons. Understand how the JobSet API addresses challenges in job orchestration, ensuring scalability and high resource utilization for heterogeneous components in cloud-native computing environments.

On-Demand Systems and Scaled Training Using the JobSet API - Abdullah Gharaibeh & Vanessa Sochat