Breaking Out of the Proprietary Cage - Real-time Data Warehouses in Open Source

Explore the world of open-source real-time data warehouses in this 51-minute Linux Foundation conference talk. Delve into the unique aspects of analytic applications and SQL data warehouses, focusing on ClickHouse as a powerful alternative to proprietary solutions. Learn about the Merge Tree table engine, its data layout, and storage optimization techniques. Discover how to boost performance through CPU parallelization, I/O optimization, and compression. Examine patterns for data management, including materialized views, TTLs, and tiered storage. Gain insights into distributed queries, Kafka-based ingestion pipelines, and visualization with Grafana. Understand ClickHouse's operation on Kubernetes and its competitive edge against proprietary SQL data warehouses in head-to-head comparisons.

Intro
Presenter Bio
What makes analytic applications special?
SQL data warehouses run analytic queries
What ClickHouse is not
Merge Tree is the workhorse table engine
Merge Tree data layout
Detailed storage layout within a single part /var/lib/clickhouse/data/airline/ontime
Adding CPUs boosts parallelized execution
I/O drives ClickHouse performance
Compression and codecs reduce I/O
Effect on storage is dramatic
Materialized views restructure/reduce data
Pattern: TTLs + downsampled views
Alternative pattern: Tiered storage
More table engines for clustering!
How do distributed queries work?
Pattern: Kafka-based ingestion pipelines
Alternative ingest pattern: Kafka engine
Pattern: Grafana visualization
Pattern: Operation on Kubernetes
Wrap-up . ClickHouse meets/beats proprietary SQL data warehouses in head-to-head comparisons