Gazelle-JNI: Offloading Spark SQL to Native Engines for Execution Acceleration

Explore Gazelle-Jni, a middle layer designed to offload Spark SQL to native engines for execution acceleration, in this 41-minute conference talk from Databricks. Learn how Gazelle-Jni implements a shared JVM and JNI middle layer to better integrate various native SQL engines as Spark SQL's backend. Discover the process of passing Substrait transformed physical plans to native engines for improved performance. Gain insights into integrating native engines with Spark SQL through practical examples. Delve into topics such as basic operator performance, CPU bottlenecks, Spark's scalability, SQL engine development, and the evolution of data processing frameworks. Examine Gluten's layout, components, plan conversion, buffer passing and sharing, fallback processing, shuffle mechanism, and memory management. Compare performance metrics for Gluten with Velox and ClickHouse. Explore future steps, learn how to take Gluten for a spin, and understand the call to action for leveraging this technology in your data processing workflows.

DATA AI
Our Team
Basic Operator Perf Stopped Grow
CPU Is The Bottleneck
Spark Is a Proved & Great Framework to Scale Out
SQL Engine Developed Years
An Evolution Is on The Way
Gluten is
Gluten Layout
Gluten Components
Plan Conversion
Buffer Passing & Sharing
Fallback Processing
Gluten's Shuffle
Gluten Memory Management
Gluten + Velox Performance
Gluten + ClickHouse Performance
Next Step
Take Gluten for a Spin
Call to Action
Performance Metrics (Velox)