Lessons Learned in Black Box Monitoring 25,000 Endpoints

Discover how the United States Digital Service tackles the challenge of monitoring 25,000 government endpoints without internal access in this 23-minute conference talk from SREcon19 Americas. Learn about black box monitoring techniques, the architecture of their monitoring system, and the implementation of tools like Prometheus Black Box Exporter and Alert Manager. Explore the process of identifying problems across government services, including case studies on Air Now.gov and USPTO. Gain insights into cost savings, results achieved, and valuable lessons learned in alerting and monitoring government systems. Understand the importance of SRE teams in government and how this approach aims to change the culture of service reliability in the public sector.

Introduction
The Digital Service
How do we find the problems
The big solution so far
Air Nowgov
USPTO
Black Box Monitoring
Government Services
MVP
Architecture
Alert Manager
What are we monitoring
Prometheus Black Box Exporter
Time Series Data
Amazon Threat
API Gateway
Cost Savings
Whats Next
Results
Chart
Lessons Learned
Alerting