Cloud-Native Apache Spark Scheduling with YuniKorn on Kubernetes
Offered By: Databricks via YouTube
Course Description
Overview
Syllabus
Intro
Role of Kos in Lyft's Data Landscape
Multi-step creation for a Spark KBs job
Problems of existing Spark K8s infrastructure Complexity of layers of custom Kås controllers to handle the scale of the
Why we need a customized K8s Scheduler
Flavors of Running Spark on KBS
Resource Scheduling in K8s
Spark on K8s: the scheduling challenges
Apache Yunikorn (Incubating)
Resource Scheduling in Yunikorn land compare w/default scheduler
Main difference (Yunikorn v.s Default Scheduler)
Run Spark with Yunikorn
Job Ordering
Resource Quota Management: K8s Namespace ResourceQuota
Resource Quota Management: Yunikorn Queue Capacity
Resource Fairness in Yunikorn Queues
Scheduler Throughput Benchmark
Fully K8s Compatible
Yunikorn Management Console
Compare Yunikorn with other K8s schedulers
Current Status
The Community
Roadmap
Our Vision - Resource Mgmt for Big Data & ML
Taught by
Databricks
Related Courses
Introduction to Artificial IntelligenceStanford University via Udacity Natural Language Processing
Columbia University via Coursera Probabilistic Graphical Models 1: Representation
Stanford University via Coursera Computer Vision: The Fundamentals
University of California, Berkeley via Coursera Learning from Data (Introductory Machine Learning course)
California Institute of Technology via Independent