Computational Semantics Evaluation - The Origins of Senseval and Evolution of SemEval

Explore the origins and evolution of computational semantics evaluation through this 56-minute lecture from the Alan Turing Institute. Delve into the history of Senseval and SemEval datasets, examining their impact on the field of computational linguistics. Discover how these evaluation exercises have shaped algorithms, data, and research communities. Investigate various semantic tasks, including word sense disambiguation, lexical substitution, and semantic textual similarity. Analyze the benefits and challenges of these evaluation methods, with a particular focus on semantic representation issues. Gain insights from Dr. Diana McCarthy, an experienced computational linguist, as she discusses the pros and cons of representation-independent tasks and their implications for interpreting results and designing future evaluations.

Intro
Outline
Computational Semantics
Evaluation Goals and Considerations
Word Sense Disambiguation (SD)
Senseval 1998
Word Sense Disambiguation: The Reality
Coarse grained Senses
Senseval 3 Panel discussion
The Evolution of SemEval
A SemEval Task Categorization
Word Sense Representation and WSI
Word Sense Induction: How to Evaluate?
Lexical Substitution (McCarthy and Navigli, 2007)
Substitutes for investigator (noun)
Logical Form Frames and Semantic Roles
Phrasal Similarity (Korkontzelos et al., 2013)
Semantic Textual Similarity (Agirre et al. 2012, 2016)
Textual Entailment
Representation may help downstream applications
Pros and cons