Scaling Monitoring at Databricks - From Prometheus to M3

Explore the journey of scaling monitoring infrastructure at Databricks, transitioning from Prometheus to M3. Learn about the initial architecture, M3 setup process, migration challenges, and valuable lessons learned. Gain insights into memory usage optimization, internal dashboard development, and strategies for handling upgrades, updates, and metric spikes. Discover capacity planning techniques and future plans for monitoring at Databricks. This comprehensive talk provides a deep dive into the complexities of large-scale monitoring systems and offers practical solutions for similar scaling challenges.

Intro
Monitoring before M3
Initial Architecture
M3 Setup
Migration
Lessons Learned
Memory Usage
Internal Dashboard
Upgrades and Updates
Metric Spikes
Capacity
Future plans
Conclusion