YoVDO

Efficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg

Offered By: The ASF via YouTube

Tags

Apache Flink Courses File Management Courses Data Processing Courses Real-Time Analytics Courses Data Streaming Courses Data Ingestion Courses Apache Iceberg Courses

Course Description

Overview

Save Big on Coursera Plus. 7,000+ courses at $160 off. Limited Time Only!
Explore the challenges and solutions for efficient, low-latency data ingestion to large tables using Apache Flink and Apache Iceberg in this 24-minute conference talk. Learn about the tradeoffs between data availability latency and optimization for efficient reading, and discover how the integration of these two Apache projects addresses these challenges. Examine the ongoing projects aimed at balancing frequent commits with optimal file management, including balanced writes and periodic compaction. Gain insights into the development process, coordination between Apache communities, and implementation details. Compare this approach with alternative solutions like Apache Hudi and Apache Paimon, understanding their pros and cons. Witness a brief demo showcasing the possibilities of this integration, presented by Marton Balassi, a Flink PMC member and Engineering Manager at Apple, and Peter Vary, an Apache Iceberg committer and Staff Engineer at Apple.

Syllabus

Efficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg


Taught by

The ASF

Related Courses

Deep Dive into Amazon Glacier
Amazon via Independent
Preparing for your Professional Data Engineer Journey
Google Cloud via Coursera
Building Resilient Streaming Systems on Google Cloud Platform en Français
Google Cloud via Coursera
IBM AI Enterprise Workflow
IBM via Coursera
Introduction to Designing Data Lakes on AWS
Amazon Web Services via edX