Evaluating Fuzz Testing

Explore the critical aspects of evaluating fuzz testing techniques in this 26-minute ACM conference talk. Delve into the general fuzzing algorithm and learn how to assess the effectiveness of new fuzzers through proper experimental setups. Discover the importance of performing multiple trials, understanding high variance, selecting appropriate seed files, and determining optimal fuzzing durations. Examine benchmark programs, methods for assessing fuzzer effectiveness, and techniques for identifying true bugs. Gain insights into using history for root cause analysis, interpreting stack hashes, and analyzing control flow when triggering bugs. Consider the potential for developing a standardized fuzzing benchmark to enhance comparability and reliability in fuzzing research.

Intro
What is fuzzing?
General fuzzing algorithm
Evaluating your new fuzzer
Performing multiple trials
High variance is real
Choice of seed file used
Timeouts: how long to fuzz?
Benchmark programs
Assessing fuzzer effectiveness
Using history to find root causes
True bugs found per run
Stack hashes
Program crashes here
Control flow when triggering bug
A fuzzing benchmark?