Improving Broadcast Joins in Apache Spark SQL
Offered By: Databricks via YouTube
Course Description
Overview
Explore the intricacies of broadcast joins in Apache Spark SQL through this 28-minute Databricks conference talk. Delve into the mechanics of Spark's execution engine, focusing on broadcast joins and their performance implications. Learn about Workday's improvements to increase the threshold for effective broadcast joins, including executor-side broadcasting and modifications to Spark's whole-stage code generator. Discover techniques for limiting memory usage in executors while increasing broadcasting thresholds. Gain insights from real-world production case studies involving large-scale ETL pipelines. Acquire valuable knowledge to optimize your own Spark workloads and enhance your understanding of Spark's join infrastructure.
Syllabus
Intro
How Spark Works
What is Broadcast Join
How Broadcast Joins Work
Improving Broadcast Joins
Single Joint
Executors
Results
Production case study
Conclusion
Taught by
Databricks
Related Courses
Building Data Engineering Pipelines in PythonDataCamp Building Your First ETL Pipeline Using Azure Databricks
Pluralsight Implementing ETL Pipelines on the Microsoft SQL Server Platform
Pluralsight Kafka Connect Fundamentals
Pluralsight DP-203 - Data Engineering on Microsoft Azure
Udemy