Incident Response: A Scientific Approach to Improving System Reliability

Explore a scientific approach to incident response in this 37-minute conference talk from Conf42 Platform Engineering 2023. Learn how safety engineering principles from other domains can be applied to improve reliability in tech systems. Discover the limitations of traditional incident response methods, including overreliance on dashboards and runbooks, guesswork, and fear of failure. Examine concepts such as normative language, mechanistic reasoning, and the theory of falsifiability. Understand why experienced troubleshooters rely on case-based strategies and how change introduces new forms of failure. Gain insights into a more hypothesis-driven approach to incident management and learn practical steps to implement this scientific methodology in your organization.

intro
preamble
incident response can learn from safety engineers in other domains
a definition...
catastrophe is always around the corner
incident response isn't easy
an overreliance of dashboards and runbooks
guesswork
spending a long time on the wrong hypothesis
fear of failure
'history doesn't repeat itselg but it often rhymes'
'it seems easy to look back at an incident and determine what went wrong ...'
normative language
mechanistic reasoning
above the line, below the line
change introduces new forms of failure
experienced troubleshootes rely more on case-based strategies
science - definition
the theory of falsifiability
'a more scientific, hypothesis-driven, approach to how humans perform ... can improve reliability
why bother?
3 steps
all practitioner acts are a gamble
thank you