Metric Perversity and Bad Decision-Making

Explore the concept of metric perversity and its impact on decision-making in security, privacy, and abuse-related fields through this 19-minute invited talk from USENIX Enigma 2023. Delve into the reasons behind poor decision-making, focusing on the role of perverse metrics that, while technically accurate, lead to undesirable outcomes. Examine the unique challenges faced in security and privacy domains, including adversarial threats, unknowns, and high-impact risks. Learn about different types of perverse metrics and discover strategies to avoid their pitfalls. Gain insights from real-world examples and understand the importance of proper metric selection in gauging performance and driving positive outcomes.

Intro
What is a metric?
Using a gauge as a lever
Adversaries don't help you measure
You didn't measure the black swans
Techniques
Good metric example