Apache Spark on Kubernetes - Lessons Learned from Launching Millions of Spark Executors
Offered By: Databricks via YouTube
Course Description
Overview
Explore lessons learned from launching millions of Spark executors on Kubernetes in this 35-minute conference talk by Databricks. Dive into Apple's approach to supporting enormous Spark workloads for cloud services, covering orchestration techniques across Mesos and Kubernetes, private and on-premise infrastructure. Learn about effective monitoring systems, resource requirement tuning, and execution analysis. Gain insights on optimizing Kubernetes for Spark workloads, implementing granular concurrency checks, mitigating cluster storage stress, and utilizing dynamic allocation. Discover strategies for push-button cloud management and scaling up Spark on Kubernetes to support varying workload patterns across multi-cloud environments.
Syllabus
Intro
Data Platform
Elastic Self Service Spark
Code to Deployment
Security
Monitoring
Orchestration Architecture
Varying Workload Pattern
One Interface over Multi-Cloud
Optimize Kubernetes for Spark Workload
Granular Concurrency Check at Orchestration
Avoid Partially Running Applications
Timeout Partially Running Applications
Mitigate Cluster Storage Stress
Utilization-based Allocation Recommendation
Dynamic Allocation
Push-button Cloud Management
Scale up Spark on Kubernetes
Taught by
Databricks
Related Courses
A Practical Guide to Amazon EKSA Cloud Guru AWS Certified Solutions Architect - Professional 2020
A Cloud Guru Azure AI Solution Requirements
A Cloud Guru Google Cloud Certified Professional Data Engineer (LA)
A Cloud Guru High Availability and Scalability for Associate AWS Solutions Architects
A Cloud Guru