Getting Started with Apache Spark on Kubernetes

Discover how to leverage Apache Spark on Kubernetes in this 26-minute video from Databricks. Learn to build, deploy, and maintain end-to-end data pipelines using cloud-agnostic technology for improved isolation and resource sharing. Explore environment setup, application sizing, performance optimization, and monitoring techniques through code-heavy demonstrations and live examples on the Data Mechanics platform. Gain valuable insights for beginners and intermediate Spark developers to successfully implement Spark on Kubernetes, covering topics such as data access, node pools, pod sizes, dynamic allocation, disk and I/O optimizations, and application logs and metrics for debugging and reporting.

Introduction
Overview
Autopilot mode
Fully containerized
Architecture
Motivations
Monitoring
Cluster Setup
Demo
Whats Next