Preventing Controller Sprawl From Taking Down Your Kubernetes Cluster

Explore the challenges and solutions surrounding controller sprawl in Kubernetes clusters through this insightful conference talk. Delve into the limitations of the widely-used WATCH and UPDATE pattern for Kubernetes controllers and understand how its overuse can lead to service degradation in production environments. Learn from real-world outages experienced by Robinhood Markets as their Container Orchestration team shares valuable lessons on monitoring, detecting, debugging, and addressing these issues. Gain practical knowledge on best practices for maintaining cluster health, including the importance of observability through metrics, logs, and audit logs. Discover the significance of change visibility and code comprehension in preventing and resolving controller-related problems. Equip yourself with essential takeaways to optimize your Kubernetes infrastructure and prevent scalability issues from impacting your cluster's performance.

Intro
Kubernetes philosophy
Kubernetes controllers
Example: Pizza controller
Case Study: #1
It's the network, stupid
Calico for L4 segmentation
Routing problem?
High ping latencies
A-HA moment!
Battle of the controllers
Why did this affect Calico?
Case study: #2
Off the charts
Theories and temporary mitigation
Do you even HTTP 410
Downward spiral
Takeaway: Observability - Metrics
Takeaway: Observability - Logs
Takeaway: Observability - Audit logs
Takeaway: Change visibility
Takeaway: Read the code!
Takeaways: Read the code!