Breaking Out of the Proprietary Cage - Real-time Data Warehouses in Open Source

Explore the world of open-source real-time data warehouses in this 51-minute Linux Foundation conference talk. Delve into the unique characteristics of analytic applications and SQL data warehouses, with a focus on ClickHouse and its Merge Tree table engine. Discover the intricacies of data layout, storage optimization, and parallelized execution. Learn about materialized views, tiered storage, and distributed queries. Examine patterns for Kafka-based ingestion pipelines, Grafana visualization, and operation on Kubernetes. Gain insights into breaking free from proprietary solutions and leveraging open-source technologies for efficient data warehousing and analytics.

Intro
What makes analytic applications special?
SQL data warehouses run analytic queries
What ClickHouse is not
Merge Tree is the workhorse table engine
Merge Tree data layout
Detailed storage layout within a single part /var/lib/clickhouse/data/airline/ontime
Adding CPUs boosts parallelized execution
Effect on storage is dramatic
Materialized views restructure/reduce data
Alternative pattern: Tiered storage
How do distributed queries work?
Pattern: Kafka-based ingestion pipelines
Alternative ingest pattern: Kafka engine
Pattern: Grafana visualization
Pattern: Operation on Kubernetes