Colocating Hadoop YARN with Kubernetes to Save Costs on Big Data
Offered By: CNCF [Cloud Native Computing Foundation] via YouTube
Course Description
Overview
Explore how to colocate Hadoop YARN with Kubernetes to significantly reduce costs on big data operations in this 43-minute conference talk by Irvin Lim and Hailin Xiang from Shopee. Learn about the challenges of low resource utilization in Kubernetes clusters and the complexities of running online services alongside offline jobs. Discover innovative solutions for customizing and extending Linux Kernel, Container Runtime, Kubernetes Scheduler, and Kubelet to improve resource utilization while maintaining the performance of online services. Gain insights into overcoming limitations of default cgroup CFS and memory limits in real-world scenarios, and understand how to navigate Kubernetes restrictions on offline job scheduling to optimize computing resource costs for big data operations.
Syllabus
Colocate Hadoop YARN with Kubernetes to Save Massive Costs on Big Data - Irvin Lim & Hailin Xiang
Taught by
CNCF [Cloud Native Computing Foundation]
Related Courses
Kubernetes Housekeeping - Resource Management and Garbage CollectionLinux Foundation via YouTube The Ins and Outs of the Cloud Provider in Kubernetes
CNCF [Cloud Native Computing Foundation] via YouTube Kubernetes SIG Node: Introduction and Deep Dive
CNCF [Cloud Native Computing Foundation] via YouTube Cgroupv2 - Coming Soon to Kubernetes Clusters
CNCF [Cloud Native Computing Foundation] via YouTube Deconstructing Kubernetes Networking - Fundamentals and Troubleshooting
CNCF [Cloud Native Computing Foundation] via YouTube