Benchmarking - You're Doing It Wrong

Discover the art of effective benchmarking in this 35-minute conference talk from Strange Loop. Explore common pitfalls and biases that can invalidate performance measurements, and learn essential techniques for creating accurate and useful benchmarks. Delve into real-world examples of flawed benchmarks, understand the impact of caches on performance testing, and gain insights into multimodal distributions. Examine the importance of focusing on critical program components and avoiding premature optimization. Uncover the challenges of microbenchmarking, non-constant work per iteration, and outlier management. Acquire valuable skills to guide your team towards meaningful performance improvements and avoid potential setbacks in system optimization.

Intro
To Write Good Benchmarks...
What's a Benchmark
Website Serving Images
It's Caches All The Way Down
Caches in Benchmarks
Wrong About the Machine
Multimodal Distribution
Wrong About Stats
Programmers waste enormous amounts of time thinking about ... the speed of noncritical parts of their programs... Forget about small efficiencies ...97% of the time
Wrong About What Matters
Becoming Less Wrong
Microbenchmarking: Blessing & Curse
Non-Constant Work Per Iteration
Outliers
Workload