Tech Talk - Top Tuning Tips for Spark 3.0 and Delta Lake on Databricks

Discover top tuning tips for Apache Spark 3.0 and Delta Lake on Databricks in this informative tech talk. Learn when to use specific join operations, how to select appropriate machine sizes, techniques to accelerate merge operations, and methods to streamline your jobs. Explore key topics including the importance of using the latest DBR version, selecting optimal join strategies, leveraging Apache Spark 3.0 and Adaptive Query Execution (AQE), partition pruning, data skipping, Z-ordering, Databricks Delta Lake and statistics, merge optimization, and choosing suitable instance types. Gain insights from experienced Databricks solutions architects and developer advocates as they share their expertise on enhancing big data processing performance and reliability using Apache Spark and Delta Lake.

Welcome
Use the latest version of DBR
Picking the best join strategy
Use Apache Spark 3.0 and AQE
Partition Pruning
Data Skipping
Z-Ordering
Databricks Delta Lake and Stats
Optimizing Merges
Picking good instance types