How the OOM-Killer Deleted My Namespace, and Other Kubernetes Tales

Explore complex Kubernetes debugging scenarios in this conference talk from KubeCon + CloudNativeCon Europe 2021. Delve into real-world examples from Datadog's journey of migrating workloads to Kubernetes, including an intriguing case where an OOM-killer invocation triggered namespace deletion. Learn about the intricate interactions between Kubernetes components, investigate symptoms, and uncover surprising root causes. Gain insights into managing large-scale clusters, understanding metrics-server setups, and dealing with CNI-related issues. Discover key takeaways about apiservice extensions, runtime troubleshooting, and the importance of informers in Kubernetes operations. Enhance your ability to diagnose and resolve unexpected challenges in Kubernetes environments.

Intro
Datadog
Symptoms
Investigation
Deletion call, 4d before Audit logs for the namespace
Spinnaker deploys (v1)
Helm 3 deploys (v2)
Big difference
What happened?
Namespace Controller logs Virtual
Events so far
Metrics-server setup
Metrics-server deployment
Full chain of events
Key take-away Apiservice extensions are great but can impact your cluster
Context
Runtime is down?
CNI status
Containerd goroutine dump Blocked goroutines?
Seems CNI related
What about Delete?
CNI plugin
The root cause
What we know
Apiserver requests
Illustration
What about label filters?
Informers instead of List How do informers work?
Back to the incident
Nodegroup controller?
How did it work?
What we learned
Conclusion