Scaling Apache Spark on Kube to Apple Scale

Explore the challenges and solutions of scaling Apache Spark on Kubernetes to Apple's massive scale in this informative conference talk. Discover which customer workloads easily ported to Apache Spark on Kubernetes and which ones faced difficulties. Learn valuable considerations and best practices for both operators and end users of Apache Spark-Kubernetes platforms. Gain insights into migrating from YARN with HDFS to Kubernetes, and understand how to effectively deploy new enhancements like shuffle tracking and graceful decommissioning. Determine when to use these features and when to avoid them. Whether you're an operator or end user, this talk will equip you with essential knowledge to optimize your Apache Spark on Kubernetes journey.

Scaling Apache Spark on Kube to Apple Scale - Amanda Moran & Holden Karau, Apple