Performance Troubleshooting Using Apache Spark Metrics - Databricks Talk

Explore performance troubleshooting techniques for Apache Spark in this 40-minute conference talk by Luca Canali from CERN. Dive into Spark's extensive metrics and instrumentation, including executor task metrics and the Dropwizard-based system. Learn how CERN's Hadoop and Spark service leverages these metrics for troubleshooting and measuring production workloads. Discover how to deploy a performance dashboard for Spark workloads and utilize sparkMeasure, a tool based on the Spark Listener interface. Gain insights into lessons learned and upcoming improvements in Apache Spark 3.0. Cover topics such as data analytics at the Large Hadron Collider, CERN's analytics platform, performance methodologies, and anti-patterns. Examine various ways to gather and analyze Spark metrics, including REST API and event logs. Explore the components of a Spark performance dashboard, including memory usage, executor CPU utilization, and user-defined metrics. Understand the importance of combining data with context to derive meaningful insights for optimizing Spark-based applications.

Intro
Data at the Large Hadron Collider
Analytics Platform @CERN
Hadoop and Spark Clusters at CERN
Performance Troubleshooting Goals
Performance Methodologies and Anti-Patterns Typical benchmark graph
Workload and Performance Data
Measuring Spark
Spark Instrumentation - Metrics
How to Gather Spark Task Metrics
Spark Metrics in REST API
Task Metrics in the Event Log
SparkMeasure - Getting Started
SparkMeasure, Usage Modes
Instrument Code with Spark Measure
Spark Metrics System • Spark is also instrumented using the Dropwizard/Codahale metrics library • Multiple sources (data providers)
Ingredients for a Spark Performance Dashboard
Assemble Dashboard Components
Spark Dashboard - Examples Graph: "number of active tasks" vs. time
Dashboard - Memory
Dashboard - Executor CPU Utilization Graph: "CPU utilization by executors' JVM" vs. time
Executor Plugins Extend Metrics • User-defined executor metrics, SPARK-28091, target Spark 3.0.0
Metrics from OS Monitoring
Data + Context = Insights