YoVDO

Accelerating Apache Spark Shuffle for Data Analytics on Cloud with Remote Persistent Memory Pools

Offered By: Databricks via YouTube

Tags

Apache Spark Courses Big Data Courses Cloud Computing Courses Distributed Systems Courses Data Analytics Courses Scalability Courses Persistent Memory Courses

Course Description

Overview

Explore a 33-minute conference talk on accelerating Apache Spark shuffle operations for cloud-based data analytics using remote persistent memory pools. Dive into the challenges of serving growing data-driven AI and analytics workloads in disaggregated storage and compute environments. Learn about a proposed fully disaggregated shuffle solution leveraging persistent memory and RDMA technologies, including a new pluggable shuffle manager and distributed storage system. Discover how this innovative approach improves Spark's scalability, performance, and reliability, with experimental results showing up to 10x performance speedup over traditional shuffle solutions. Gain insights into the architecture, optimization features, and workflow of this cutting-edge solution presented by Databricks.

Syllabus

Introduction
Agenda
Motivation
Recap
Original Example
Results
New Challenges
Rtmp Architecture
Optimization Features
Workflow
Summary
Performance Evaluation
Examples
Call to Action
Optima Natives


Taught by

Databricks

Related Courses

Stanford Seminar - The Quest for Low Storage Latency Changes Everything
Stanford University via YouTube
Promise and Pitfalls of Persistent Memory
Strange Loop Conference via YouTube
Crimson - A New Ceph OSD for the Age of Persistent Memory and Fast NVMe Storage
USENIX via YouTube
Write-Optimized Dynamic Hashing for Persistent Memory
USENIX via YouTube
ctFS - Replacing File Indexing with Hardware Memory Translation through Contiguous File Allocation for Persistent Memory
USENIX via YouTube