Running Apache Spark on Kubernetes - Best Practices and Pitfalls

Explore best practices and potential pitfalls of running Apache Spark on Kubernetes in this 25-minute conference talk from Databricks. Dive into core concepts, setup procedures, and configuration tips for optimizing performance and resource sharing. Learn about Spark-app level dynamic allocation, cluster level autoscaling, and Kubernetes-specific considerations for data I/O performance. Discover monitoring and security best practices, as well as current limitations and planned future developments. Gain valuable insights from lessons learned while building a serverless Spark platform powered by Kubernetes, covering topics such as efficient resource usage, spot instance management, and security measures. Conclude with a high-level checklist to ensure successful implementation of Spark on Kubernetes in your data analytics infrastructure.

Intro
About Data Mechanics
Core Concepts
Configuration Tips
Spark Performance
Pod Resource Usage Manager
Spot History Server
Timeseries DB
Security
Upcoming features
Conclusion
Highlevel checklist