The Abuse Uncertainty Principle, and Other Lessons Learned from Measuring Abuse on the Internet - David Freeman - USENIX Enigma Conference - 2020

Explore the challenges and strategies of measuring and combating online abuse in this 27-minute conference talk from USENIX Enigma 2020. Delve into Facebook's approach to fighting spam, phishing, and other forms of internet abuse as David Freeman presents the "Abuse Uncertainty Principle" and discusses various measurement techniques. Learn about user reports, human labeling, and automated labeling methods, and understand their appropriate applications. Discover how effective abuse measurement can drive prioritization and impact analysis in integrity teams. Gain insights into determining background levels of spam and fake accounts, assessing detection system effectiveness, and identifying critical abuse problems. Examine the implications of the Abuse Uncertainty Principle on metric usage and explore strategies for developing reliable metrics to inform key decisions. Apply these tools to evaluate abuse levels in your product, assess prevention effectiveness, and determine areas for future investment.

Intro
Sometimes I wish I worked on ad ranking...
Abusive content on Facebook
Measuring abuse is hard!
Measuring abuse: The Dark Ages
Measuring Abuse: The Renaissance
Benefits of Measurement (in addition to a quieter office)
How do we get there?
Count what you blocked These things have to be bad, right?
Have users do the labeling Crowdsource the work via reporting/appeals flows.
Human labeling Get experts to decide for you.
Deployment at Facebook Used for internal direction and/or external reporting of
Deployment at Facebook Used for internal direction on
Comparison of approaches
Example: Rate Limiting
"Proof" of Goodhart's Law
Corollary: "Abuse Uncertainty Principle"
Living with the Uncertainty Principle For spam detection at Facebook, we split signals into two classes
Other coping mechanisms
Challenge: Build an Abuse Adapter!
Open questions
Thank you!