Managing Multi-Cloud Apache Spark on Kubernetes

Explore the challenges and solutions of managing multi-cloud Apache Spark on Kubernetes in this 31-minute conference talk by Ilan Filonenko and Aki Sukegawa from Bloomberg. Dive into Bloomberg's journey of building multi-cloud quant platforms on Kubernetes for financial applications with integrated data science capabilities. Learn about the complexities of managing data science infrastructure across multiple cloud environments, focusing on Apache Spark. Discover strategies for effective Spark infrastructure management spanning bare-metal and public cloud platforms. Examine approaches to auto-scaling, scheduling, preemption, and security in Kubernetes. Gain insights into observability techniques, including methods to surface cluster information to diverse Spark end-users using native Kubernetes resources such as node autoscalers, controllers, and custom PodConditions. Follow the speakers as they discuss user stories, complications, and solutions, exploring topics like custom resources, cluster scaling, event handling, and PodStatus Controller behavior.

Intro
Background (Kubernetes)
Background (Apache Spark)
Background (Spark)
User Stories (Complications)
User Stories (Solutions)
Why custom resource (CR)
Storing information, where to?
First: Cluster scaling up
Cluster autoscaler events
Controller to look up event objects
Next: Scaling down, OOM, etc.
Keeping pods?
Kubernetes custom resource (CR)
PodStatus Controller behavior
Extra: Declarative copying
Extensions