Troubleshooting Tiered Tragedy - A Peek Into Failure

Explore a real-world failure in a distributed system and the troubleshooting process involved in this 42-minute conference talk from GOTO Chicago 2017. Follow Jeff Smith, Manager of Production Operations at Centro, as he dissects the anatomy of a system, defines failure modes, and walks through the signs of trouble. Learn about shutting down systems, identifying changes, rolling back, and uncovering missed issues. Gain insights on the importance of sensible defaults, contextual metrics, centralized logging, and alert management. Conclude with a recap and Q&A session to deepen your understanding of handling complex system failures in DevOps environments.

Introduction
Anatomy of a System
System Definition
Failure Modes
Failure Walkthrough
Signs of Trouble
Shutting It Down
What Changed
Roll It Back
What We Missed
A sensible default
What we learned
Metrics need context
Centralized logging
Losing alerts
Recap
Questions