Ground Truth - The Uses and Abuses of Human Annotation in Text Analysis - Professor Kenneth Benoit

Explore the concept of "ground truth" in text analysis and its implications for social and cultural research in this 40-minute talk by Professor Kenneth Benoit at the Alan Turing Institute. Delve into the challenges of human annotation, examining sources of ground truth and their applications in various text analysis scenarios. Investigate methods for validating latent positions and learn about recoding experiments and intercoder agreement. Discover recent developments in text annotation using crowdsourcing, with examples from immigration and economic policy studies. Compare expert coding to crowd-sourced annotations, and understand the importance of quality control in these processes. Gain insights into the uses and potential abuses of human annotation in text analysis, bridging the gap between NLP/ML methodologies and humanities and social sciences research.

Intro
the curious notion of "ground truth"
the curious notion of ground truth
sources of ground truth
examples in text analysis
sometimes it's far from obvious
how do we validate latent positions?
a recoding experiment
intercoder agreement
misclassification
more recent investigations
text annotation using the crowd
example: labelling immigration sentences
example: economic and social policy
start with expert coding, for benchmark
deployment on Crowdflower
inter-coder agreement
comparing experts to crowd-coders
correlation of aggregate measures
experts were just a less variable crowd
example: immigration policy
test for immigration policy
quality control: very important
conclusions