Unified and Pluggable Cluster Management Platform at LinkedIn

Explore LinkedIn's journey in adopting Kubernetes for their next-generation cluster management system in this conference talk. Learn about the challenges faced and lessons learned during the integration process with LinkedIn's existing ecosystem. Discover how the company evaluated and decided to adopt Kubernetes' declarative API and extensible architecture to support heterogeneous compute workloads. Gain insights into LinkedIn's cluster management ecosystem, including RAIN, their resource management system. Understand the performance and extensibility challenges encountered with Kubernetes, and how LinkedIn addressed them through a pluggable scheduler architecture. Examine specific use cases such as Jupiter Hub, batch processing, and model serving. Compare the Stack vs Kubernetes approach and learn how LinkedIn aims to bring both worlds together. Delve into topics like Kerberos integration and unified topology as part of their adaptation process.

Introduction
Agenda
About LinkedIn
Our Mission
Our Scale
Our Cost Management Platform
Fabric Architecture
Deployment Control Plane
Locker Container
Batch Jobs
Challenges
Kubernetes Performance
Kubernetes extensibility
Pluggable scheduler architecture
First use case
Jupiter Hub
Kubernetes Journey
Notebook Pod Crash
Debugging
Particle Networking
Pod Network
Pod Network Crash
Jupiter Hub Pod
Pod Preset
Batch Batch
Model Serving
Use Cases
Stack vs Kubernetes
Bringing both worlds together
Kerberos
Unified Topology
Wrap Up