Fugue: Unifying Big Data Analytics Ecosystems for ETL and Machine Learning

Explore the Fugue framework, an abstraction layer unifying various big data analytics solutions like Apache Spark, TensorFlow, Druid, Dask, and Flink. Learn how this SQL-like language represents end-to-end pipelines, extensible with Python, to create reliable, performant, and maintainable data processing workflows. Discover the benefits of a unified K8S Spark environment for interactive development, batch processing, and near real-time streaming jobs. See demonstrations of instant dependency updates, on-demand Spark K8s cluster management, and Fugue extensions for Kinesis and Kafka. Understand how Fugue provides abstraction for machine learning pipelines, enabling distributed training, hyperparameter tuning, and inference across various ML libraries. Gain insights into extensive testing on Spark 3.0 and the resulting performance improvements in this 22-minute talk from Databricks.

Intro
Motivation of Fugue
Node Vec: Fugue Code
Fugue Programming Model
A Workflow Example
The Fugue Extensions
Fugue SQL vs Spark SQL
Fugue Programming Interface vs SQL
Fugue ML Components
Model & Parameter Sweeping model
Benchmark Test
An Interactive On-demand Spark Ecosystem
Summary