Apache Spark on Kubernetes - A Technical Deep Dive

Dive into a technical deep dive conference talk exploring Apache Spark's integration with Kubernetes. Discover the evolution of Spark's scheduler backend, which now includes native Kubernetes support as of version 2.3.0. Explore the technical intricacies of this integration, including the Kubernetes submission client, scheduler backend, and how executors connect to drivers. Learn about dependency management, existing features like client mode and Kerberos support, and the Kubernetes Operator for Spark. Gain insights into the project's history, current capabilities, and future roadmap. Understand why running Spark on Kubernetes is beneficial and how it leverages Kubernetes for scheduling Spark drivers and executors. Get involved in this exciting development at the intersection of big data processing and cloud-native technologies.

Intro
Why Spark on Kubernetes?
Apache Spark
Brief Project History
Spark Meets Kubernetes
Kubernetes Submission Client
Kubernetes Scheduler Backend
Under the Hood
Executors Connecting to Driver
Dependency Management
Existing Features
Client Mode (2.4)
Kerberos Support (3.0)
Kubernetes Operator for Spark
Roadmap (3.0 and Beyond)
Getting involved