Degrading Performance - Understanding and Solving Small Files Syndrome
Offered By: Databricks via YouTube
Course Description
Overview
Explore the impact of small files on data pipeline performance in this 27-minute video from Databricks. Learn about the challenges posed by small files, particularly in Parquet format, across real-time, near-real-time, and batch processing jobs. Understand how cloud storage interacts with Apache Spark, delve into Parquet and Storage API calls, and discover why small files can degrade system performance. Gain insights into leveraging DeltaLake for a more efficient solution. Follow along with a comprehensive agenda covering Spark query lifecycle abstraction, Azure Data Lake Gen 2, file hierarchy detection, and a practical demonstration.
Syllabus
Intro
Agenda
Spark query lifecycle abstraction
Azure Data Lake Gen 2
Detecting
File Hierarchy
Demo
Demo Summary
Outro
Taught by
Databricks
Related Courses
Coding the Matrix: Linear Algebra through Computer Science ApplicationsBrown University via Coursera كيف تفكر الآلات - مقدمة في تقنيات الحوسبة
King Fahd University of Petroleum and Minerals via Rwaq (رواق) Datascience et Analyse situationnelle : dans les coulisses du Big Data
IONIS via IONIS Data Lakes for Big Data
EdCast 統計学Ⅰ:データ分析の基礎 (ga014)
University of Tokyo via gacco