Data-Driven or Data-Blinded? Uses and Abuses of Analytics in Games

Explore common mistakes and pitfalls in game data analysis through this 2018 GDC conference talk. Gain practical guidance on improving test rigor and data quality as Kongregate's Emily Greer delves into topics such as audience mix, sample sizes, outliers, correlation pitfalls, A/B testing, and statistical significance. Learn about the game data lifecycle, creative iteration, and pre-production testing while understanding that there's no single right answer in data-driven game development.

Intro
Data is a Hot Mess
A Tale of Two Games
A Different Tale of Two Games
Triangulating Truth
Building Worlds
Audience Mix
Audience Age
Averages are Average Tutorial Completion Rate
Small Sample Sizes
Nothing is Normal
Outliers Rule
Cherry Picking
Confirmation Bias
Axis of Evil
Everything Correlates with Engagement
A/B Tests
(Miss) Assignment
(Huge Miss) Assignment
Downstream Example
Premature Analysis
Statistical Significance True Mean
Abnormal Testing
Medians and Distribution FTW
Extreme Description Testing
Meaningful No Difference
A/B/C(ontext) Testing
Not Everything is Testable
Hierarchy of Testing
Game Data Lifecycle
Creative Iteration
Pre-Production Testing
There's No Right Answer