Degrading Performance - Understanding and Solving Small Files Syndrome
Offered By: Databricks via YouTube
Course Description
Overview
Explore the impact of small files on data pipeline performance in this 27-minute video from Databricks. Learn about the challenges posed by small files, particularly in Parquet format, across real-time, near-real-time, and batch processing jobs. Understand how cloud storage interacts with Apache Spark, delve into Parquet and Storage API calls, and discover why small files can degrade system performance. Gain insights into leveraging DeltaLake for a more efficient solution. Follow along with a comprehensive agenda covering Spark query lifecycle abstraction, Azure Data Lake Gen 2, file hierarchy detection, and a practical demonstration.
Syllabus
Intro
Agenda
Spark query lifecycle abstraction
Azure Data Lake Gen 2
Detecting
File Hierarchy
Demo
Demo Summary
Outro
Taught by
Databricks
Related Courses
CS115x: Advanced Apache Spark for Data Science and Data EngineeringUniversity of California, Berkeley via edX Big Data Analytics
University of Adelaide via edX Big Data Essentials: HDFS, MapReduce and Spark RDD
Yandex via Coursera Big Data Analysis: Hive, Spark SQL, DataFrames and GraphFrames
Yandex via Coursera Introduction to Apache Spark and AWS
University of London International Programmes via Coursera