The Meaning and Measurement of Bias - Lessons from NLP

Explore a comprehensive tutorial from FAT*2020 that delves into the complex world of bias in Natural Language Processing (NLP). Gain insights from experts Abigail Jacobs, Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach as they unpack the meaning and measurement of bias. Discover the language and tools of construct validity, examining real-world examples from measuring height to evaluating recidivism prediction models. Investigate how fairness is conceptualized as an unobserved theoretical construct and learn about precise mathematical definitions. Analyze the intersection of measurement modeling and NLP, with a focus on representational harms in NLP systems and word embeddings. Enhance your understanding of bias measurement in AI and its implications for fairness in machine learning applications.

Intro
Things we care about
The measurement process
Cartoon of a ML pipeline
This tutorial We introduce the language and the tools of construct validity
Measuring height
Measuring socioeconomic status using income
Measuring topics using word counts
Evaluating measurement models
Recidivism
Fairness is an unobserved theoretical construct
Measuring faimess' Precise mathematical definitions of timess
Measurement is everywhere
Measurement modeling and NLP
Measuring "bias" in NLP systems
Potential constructs of interest: Representational harms from NLP systems
Example 1: Word embeddings