AB-Tests, Lies, Damned Lies, and Statistics

Explore the world of data-driven decision-making in software development through this insightful conference talk. Delve into AB-testing methodologies, learn to create and implement effective tests within your codebase, and discover how to determine the significance of changes. Examine the balance between qualitative user testing and data-driven approaches, and understand their roles in your AB-testing strategy. Investigate potential pitfalls of data-driven development, including inaccurate data, and learn strategies for validating information to ensure confident decision-making. Gain practical knowledge on topics such as randomness, biases, KPIs, user journeys, and statistical analysis. Explore tools like Optimizely and Google Analytics for implementing and tracking tests. Discover techniques for avoiding common mistakes, such as focusing on wrong metrics or misinterpreting data. Learn how to quantify subjective elements, remain context-aware, and cultivate curiosity in your development process. By the end of this talk, you'll be equipped with valuable insights to enhance your software development approach and make more informed, data-backed decisions.

Introduction
Who am I
ABtestdriven development
Doing the right thing
Biases
ABtest
Randomness
Questioning
Making better decisions
KPIs
Page Views
User Testing
Optimizely
Statistics
User Journey
Learning to make better decisions
The naive approach
Dividing concrete
Buckets
Bucket ID
Bucket Cookie
Bucket IDs
Google Analytics
Dashboards
histogram
tracking all tests
red button test
negative testing
nobody uses the data
focus on the wrong metrics
name your data
sampling
significance metrics
broken code
proxy testing
browser mode proxy
selenium
curve for change
double verification
subjectivity
quantify everything
be context aware
feel your curiosity
testdriven development
express development