How to Fail at Benchmarking

Explore the intricacies of effective benchmarking in this 54-minute Devoxx conference talk. Delve into crucial metrics for measurement, analyze the significance of average latency, and uncover the Coordinated Omission issue. Learn to identify common pitfalls in benchmarking, understand the importance of various factors like caches, clock resolution, and background noise, and gain insights into statistical concepts such as normal distribution and HDR histograms. Discover practical techniques for measuring response time, setting appropriate expectations, and interpreting results accurately. Equip yourself with the knowledge to conduct meaningful benchmarks, avoid misleading conclusions, and make informed decisions based on reliable performance data.

Introduction
Cassandra
Datastax
Benchmarks
Squid vs Cat
Why this talk
Benchmarking is a people exercise
We have caches everywhere
On our infrastructure we have
Database
Clock resolution
Warming period
Background noise
Windows XP
Average
Capacity
Average Latency
Response Time
How many 9s
Unluckiest users
Measuring response time
Requirements
The good part
Statistics
Yellow Lab Tools
Normal Distribution
Un unskilled
Metrics
HDR histogram
Graphing
Code
Flag
Explanation
Prerun check
Benchmark throughput
Benchmark latency
Benchmark expectations
Outliers
Results
Reality
Latency
Coordinated Emission
Test the tester
The request should have been sent
Another fix
Summary