The Data Science of Experimental Design

Learn how to design an A/B test for a web page, run the test, analyze the data, and make decisions based on the results of the test.

Introduction

• Conducting an experiment online
• What you should know

1. Introduction to Experimental Testing

• Experiment is a type of study
• Features of an experiment
• Circumstances for experimental testing
• When not to do an experiment
• Systems ready for experimental testing
• Comparability of experimental conditions

2. Defining Conversions

• Trying to increase conversions
• Different types of conversions
• Case definition of conversion
• Measuring a conversion
• Considering time period for conversions
• Rates versus frequencies of conversions

3. Defining Conversion Rates

• Identify and prioritize conversions
• Operationalize counting conversions
• Document conversion case definitions
• Brainstorm denominators
• False positives and negatives
• Document denominators
• Determine time frames

4. Baseline Descriptive Analyses

• Baseline time-series analyses
• Data handling
• Baseline results as a guide
• Thinking about increasing conversions
• Strategies to increase conversions
• Planning a campaign

5. Designing the Experiment

• Designing the test
• Testing the implementation
• Choosing a test statistic
• Choosing the chi-squared test
• Chi-squared test in Excel

6. Sample Size and Statistics

• Installing G*Power
• Using G*Power
• Sample size simulation
• Planning the timeline
• Stratified analysis
• Conditional tests

7. Analyzing and Interpreting the Data

• Overall analysis approach
• Time-series analysis
• Chi-squared analysis
• Interpretation

Conclusion and Next Steps

• What actions can we take?
• Report writing