How We are Dealing with Metrics at Scale on GitLab.com

Explore how GitLab.com tackles the challenge of managing metrics at scale in this 20-minute conference talk by Andrew Newdigate. Discover the innovative approach to cataloging key metrics for each application component, leading to automated generation of Grafana dashboards, robust alerting rules, and high-quality SLA indicators. Learn about the evolution from drowning in data during incidents to implementing a streamlined system for effective monitoring. Gain insights into dealing with multiple silos, split brain alerts, and the use of Thanos Rule. While primarily focused on Prometheus users, the fundamental concepts presented can be applied to various metrics systems. Dive into topics such as key metrics, alerts, metrics catalogs, and example configurations to enhance your understanding of large-scale metrics management.

Intro
Key Metrics
Alerts
Metrics Catalog
Simplest Approach
Multiple Silos
Split Brain Alerts
Thanos Rule
Example Configuration
Conclusion
Resources