Scaling OpenTelemetry Collectors Using Kafka - Conf42 SRE 2024

Explore a conference talk on scaling OpenTelemetry collectors using Kafka, presented by Pranay Prateek at Conf42 SRE 2024. Dive into the world of observability platforms, focusing on SigNoz, an open-source solution. Learn about OpenTelemetry's importance and its collector architecture. Examine the challenges of scaling with OpenTelemetry collectors alone and discover how integrating Kafka can address these issues. Gain insights into Kafka setup, monitoring consumer lag, scaling strategies, and producer-consumer latency. Understand the benefits of a Kafka-based architecture and potential areas for improvement. Get involved in the growing community and expand your knowledge of modern observability solutions for site reliability engineering.

intro
preamble
about me
signoz - open source observability platform
what is opentelemetry?
why is opentelemetry important?
introduction to opentelemetry collector
architecture of signoz cloud single tenant without kafka
issues with scaling with just opentelemetry collector
architecture of signoz cloud with kafka
how kafka can help
kafka setup, records
monitoring consumer lag is important
scaling based on consumer lag
monitoring producer - consumer latency
kafka based architecture is working well so far...
potential improvements
get involved in a growing community
thank you