Datalake Rock Paper Scissors: Iceberg with Flink or Spark - Performance Comparison

Explore a conference talk from Current 2023 comparing Apache Flink and Apache Spark for ingesting data from Apache Kafka into an Apache Iceberg datalake. Learn from Bloomberg's experiences as Sitarama Chekuri and Ben de Vera share insights on functionality, performance, fault-tolerance, scaling, and resource utilization of both technologies. Gain valuable knowledge about real-time data pipelines and storage sinks, with a focus on near-real-time speeds. Discover the motivations behind Bloomberg's approach, get an overview of the technologies involved, and examine performance comparisons. Understand how to scale to multiple applications and benefit from the speakers' summary of lessons learned. This 36-minute presentation provides a comprehensive look at datalake architecture choices for organizations using Kafka and Iceberg in their data infrastructure.

- Intro
- Context on Bloomberg and speakers
- Motivation
- Technology overview
- Performance comparison
- Scale to multiple applications
- Summary