Stream vs. Batch - Leveraging M3 and Thanos for Real-Time Aggregation

Explore the ongoing debate between stream and batch processing in this 19-minute conference talk from KubeCon + CloudNativeCon Europe 2022. Dive into the challenges of monitoring business-critical applications at scale and discover how to process large volumes of real-time data while maintaining valuable insights. Learn about two popular open-source projects, M3 and Thanos, and their approaches to real-time aggregation. Examine the methodologies leveraged by the community to aggregate data in real-time, including streaming and batch processing, and understand the tradeoffs of each approach. Gain insights into high cardinality metrics, querying without aggregation, Prometheus recording rules, and the pros and cons of streaming and batch aggregation. By the end of the talk, acquire the knowledge to make informed decisions on choosing the right aggregation method for your cloud-native applications.

Intro
High Cardinality Metrics Example
Querying without aggregation
Prometheus Recording Rules
Open source metrics solutions
What is M3?
Streaming aggregation with M3
Pros and cons of streaming aggregation
What is Thanos?
Batch aggregation with Thanos
Pros and cons of batch aggregation
Recap - How to choose?