Don't Page Me! How We Limit Pager Noise at New Relic

Discover effective strategies for minimizing pager noise in high-traffic systems through this RailsConf 2022 talk. Learn how New Relic manages its largest monolith, handling 200,000 requests per minute and interacting with numerous external services and databases, while maintaining stability. Explore a successful process for identifying trustworthy data, refining alert conditions, and determining which checks warrant paging. Gain insights into implementing heartbeat monitors, integration tests, and other techniques to ensure system reliability without constant interruptions. Apply these lessons to improve your own alerting systems and maintain a healthier work-life balance for your on-call team.

Intro
Overview
Alerts
Alert Conditions
Example
Is this a big company problem
Heartbeat Monitors
Integration Tests