Kubernetes DNS Horror Stories - And How to Avoid Them

Explore DNS challenges and solutions in large-scale Kubernetes environments through this informative conference talk. Dive into the intricacies of Kubernetes DNS, including its default configuration and access methods. Learn about unexpected issues encountered by Datadog while managing clusters with thousands of nodes and high DNS query loads. Discover how to handle problems related to IPv6, Go implementations, CoreDNS Autopath, and node-local DNS. Gain insights into mitigating issues such as source port reuse, OOM-killed CoreDNS instances, and upstream DNS problems. Understand the importance of proper DNS setup and monitoring in Kubernetes deployments, and acquire strategies to detect and resolve DNS-related issues before they impact your system's performance and reliability.

Intro
Datadog
Challenges
What we did not expect
Outline
How it works (by default)
Accessing DNS
Theory: Scenario 2
In practice
Let's disable IPv6!
What triggers IPv6?
We use Go a lot
Subtle difference
CGO implementation
Coredns Autopath
Node-local-dns
Initial state
Pod A deleted
Source port reuse
Mitigation #2
Coredns getting OOM-killed
Proportional autoscaler
Staging fright on AWS
Upstream DNS issue
Something weird
DNS is broken for a single app
Sometimes it's not DNS
Our DNS setup
Conclusion • Running Kubernetes means running DNS DNS is hard, especially at scale