Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors
Offered By: Databricks via YouTube
Course Description
Overview
Explore lessons learned from launching millions of Spark executors on Kubernetes in this 35-minute conference talk by Databricks. Dive into Apple's approach to supporting enormous Spark workloads for cloud services, covering orchestration techniques across Mesos and Kubernetes, private and on-premise infrastructure. Learn about effective monitoring systems, resource requirement tuning, and execution analysis. Gain insights on optimizing Kubernetes for Spark workloads, implementing granular concurrency checks, mitigating cluster storage stress, and utilizing dynamic allocation. Discover strategies for push-button cloud management and scaling up Spark on Kubernetes to support varying workload patterns across multi-cloud environments.
Syllabus
Intro
Data Platform
Elastic Self Service Spark
Code to Deployment
Security
Monitoring
Orchestration Architecture
Varying Workload Pattern
One Interface over Multi-Cloud
Optimize Kubernetes for Spark Workload
Granular Concurrency Check at Orchestration
Avoid Partially Running Applications
Timeout Partially Running Applications
Mitigate Cluster Storage Stress
Utilization-based Allocation Recommendation
Dynamic Allocation
Push-button Cloud Management
Scale up Spark on Kubernetes
Taught by
Databricks
Related Courses
Cloud Computing Concepts, Part 1University of Illinois at Urbana-Champaign via Coursera Cloud Computing Concepts: Part 2
University of Illinois at Urbana-Champaign via Coursera Reliable Distributed Algorithms - Part 1
KTH Royal Institute of Technology via edX Introduction to Apache Spark and AWS
University of London International Programmes via Coursera Réalisez des calculs distribués sur des données massives
CentraleSupélec via OpenClassrooms