YoVDO

Degrading Performance - Understanding and Solving Small Files Syndrome

Offered By: Databricks via YouTube

Tags

Apache Spark Courses Data Processing Courses Cloud Storage Courses Parquet Courses

Course Description

Overview

Explore the impact of small files on data pipeline performance in this 27-minute video from Databricks. Learn about the challenges posed by small files, particularly in Parquet format, across real-time, near-real-time, and batch processing jobs. Understand how cloud storage interacts with Apache Spark, delve into Parquet and Storage API calls, and discover why small files can degrade system performance. Gain insights into leveraging DeltaLake for a more efficient solution. Follow along with a comprehensive agenda covering Spark query lifecycle abstraction, Azure Data Lake Gen 2, file hierarchy detection, and a practical demonstration.

Syllabus

Intro
Agenda
Spark query lifecycle abstraction
Azure Data Lake Gen 2
Detecting
File Hierarchy
Demo
Demo Summary
Outro


Taught by

Databricks

Related Courses

Architecting Microsoft Azure Solutions
Microsoft via edX
Computing, Storage and Security with Google Cloud Platform
Google via Coursera
Windows Server 2016: Azure for On-Premises Administrators
Microsoft via edX
Microsoft Professional Orientation : Cloud Administration
Microsoft via edX
IT Support: Troubleshooting Microsoft Office
Microsoft via edX