User Defined Aggregation in Apache Spark - From Challenges to Improvements

Explore the evolution and power of User Defined Aggregate Functions (UDAFs) in Apache Spark through this 21-minute Databricks conference talk. Delve into the journey of customized scalable aggregation logic, from its initial flaws to the improved design in Spark 3.0. Learn how to create your own UDAF library, understand the inner workings of User Defined Aggregation, and discover how the latest UDAF features enhance both usability and performance. Gain insights into the Apache Spark code review process and pick up valuable tips for successfully integrating large features into the upstream community. Follow the speaker's personal experience with UDAFs, from initial challenges to ultimate triumph, while acquiring practical knowledge about this powerful feature in Apache Spark's data processing capabilities.

Intro
Spark's Scale-Out World
Scale-Out Sum
Spark Aggregators
Data Sketching: T-Digest
Is T-Digest an Aggregator?
Romantic Chemistry
Romantic Montage
UDAF Anatomy
What Could Go Wrong?
Wait What?
SPARK-27296
Aggregator Anatomy
Intuitive Serialization
Custom Aggregation in Spark 3.0
Performance
Don't Give Up
Patience
Respect