Lesson Learned on Running Hadoop on Kubernetes

Explore lessons learned from running Hadoop on Kubernetes in this insightful conference talk by Chen Qiang from LinkedIn. Discover how LinkedIn, operating one of the world's largest Hadoop environments, leveraged Kubernetes to streamline testing of cluster features. Learn about the challenges faced in traditional testing environments and how deploying production-like Hadoop clusters on Kubernetes significantly reduced infrastructure setup time from weeks to minutes. Gain valuable insights into overcoming network challenges, keytab delivery, and orchestration issues. The presentation covers the benefits of this approach, including improved isolation for testing new features and eliminating hardware dependencies. Conclude with a demonstration and key takeaways for implementing Hadoop on Kubernetes in large-scale environments.

Introduction
What is Hadoop
Hadoop at LinkedIn
Why Hadoop on Kubernetes
Kubernetes Overlay Network
Challenges
Network
Keytab Delivery
Orchestration
What is next
Key takeaway
Demo