A/B Testing in Python

Learn the practical uses of A/B testing in Python to run and analyze experiments. Master p-values, sanity checks, and analysis to guide business decisions.

In this course, you will dive into A/B testing by learning to design, run, and analyze these A/B tests in Python. You’ll start by learning to define the right metrics before estimating the appropriate sample size and duration to yield conclusive results. Throughout this course, you’ll use a range of Python packages to help with A/B testing, including statsmodels, scipy, and pingouin. By the end, you will be able to run checks that guarantee accurate results, master the art of p-values, and analyze the results of A/B tests with ease and confidence to guide the most critical business decisions.

• Overview of A/B Testing
• In this chapter, you’ll learn the foundations of A/B testing. You’ll explore clear steps and use cases, learn the reasons and value of designing and running A/B tests, and discover the most commonly used metrics design and estimation frameworks.
• Experiment Design and Planning
• In Chapter 2, you’ll cover the experiment design process. Starting with learning how to formulate strong A/B testing hypotheses, you’ll also cover statistical concepts such as power, error rates, and minimum detectable effects. You’ll finish the chapter by learning to estimate the appropriate sample size needed to yield conclusive results and tackle scenarios with multiple comparisons.
• Data Processing, Sanity Checks, and Results Analysis
• Here, you’ll discover a concrete workflow for cleaning, preprocessing, and exploring AB testing data, as well as learn the necessary sanity checks we need to follow to ensure valid results. You’ll explore a detailed explanation and example of analyzing difference in proportions A/B tests.
• Practical Considerations and Making Decisions
• In the final chapter, you’ll develop frameworks for analyzing differences in means and leveraging non-parametric tests when several assumptions aren't met. You’ll also learn how to apply the Delta method when analyzing ratio metrics and discover the best practices and some advanced topics to continue the A/B testing mastery journey.