How to A/B Test with Confidence

Learn how to conduct reliable A/B tests in this 33-minute conference talk from Ruby Central. Explore the fundamentals of A/B testing, including significance, sample size, and group randomization. Discover common pitfalls in setup, implementation, and analysis that can skew results. Gain actionable tips to mitigate issues and ensure accurate testing. Cover topics such as avoiding short experiments, agreeing on setup, validating significance calculations, and proper documentation. Equip yourself with a comprehensive checklist to confidently make data-driven decisions for your app.

Introduction
The plan
What is an AB test
Is this difference real
What is significance
How small a change
Sample size
Beware of short experiments
Bayesian AB test
Group randomisation
Nonrandom split
Starting too early
Not agreeing the setup
Accidental differences
Validate significance calculation
What can go wrong
Manufacturing results
Not neutral
Fixing bugs
Documentation
Checklist