YoVDO

Caching Framework for Exabyte-Scale Data Lakes

Offered By: The ASF via YouTube

Tags

Data Lakes Courses Hadoop Courses Cloud Migration Courses Parquet Courses Alluxio Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Discover how to design and implement an open-source caching framework for exabyte-scale data lakes in this 40-minute conference talk. Learn about the challenges of data access performance and cost in large-scale data lakes, and explore solutions to improve performance by 1.5x while reducing storage costs by millions per year. Gain insights into architecting caching systems that accelerate queries, maximize cache hit rates, and cut data storage costs. Explore the open-source stack, including Hadoop, Parquet, Hudi, and Alluxio, used to achieve a balance between performance and cost efficiency. Delve into advanced techniques for improving cache hit rates, such as segmented data file caching, soft-affinity scheduler policies, and cache filtering. Learn how to monitor cache usage and working set size using comprehensive trace and JMX metrics. By the end of this session, acquire valuable knowledge to tackle the complexities of managing and optimizing exabyte-scale data lakes in both on-premises and cloud environments.

Syllabus

Caching Framework for Exabyte-Scale Data Lakes


Taught by

The ASF

Related Courses

Cloud Computing Engineering and Management
University System of Maryland via edX
Migrating Workloads to Azure
Microsoft via edX
Exam Readiness: AWS Certified Solutions Architect - Professional (Digital)
Amazon via Independent
AWS Fundamentals: Migrating to the Cloud
Amazon Web Services via Coursera
Upgrade2Success – Mastering HCM Migration
SAP Learning