Neural Nets for NLP 2021 - Distributional Semantics and Word Vectors

Learn about distributional semantics and word vectors in this comprehensive lecture from CMU's Neural Networks for NLP course. Explore techniques for describing words by their context, including counting and prediction methods. Dive into skip-grams, continuous bag-of-words (CBOW), and advanced word vector approaches. Discover methods for evaluating and visualizing word vectors, and gain insights into their limitations and applications. Examine topics such as contextualization, WordNet, GloVe, intrinsic and extrinsic evaluation, pre-trained embeddings, and techniques for improving embeddings like retrofitting and de-biasing.

Intro
Remember: Neural Models
How to Train Embeddings?
What do we want to know about words?
Contextualization of Word Representations
A Manual Attempt: WordNet
An Answer (?): Word Embeddings!
Word Embeddings are Cool! (An Obligatory Slide)
Distributional vs. Distributed Representations
Distributional Representations (see Goldberg 10.4.1)
Count-based Methods
Prediction-basd Methods (See Goldberg 10.4.2)
Word Embeddings from Language Models giving
Context Window Methods
Glove (Pennington et al. 2014)
What Contexts?
Types of Evaluation
Non-linear Projection • Non-linear projections group things that are close in high
t-SNE Visualization can be Misleading! Wattenberg et al. 2016
Intrinsic Evaluation of Embeddings (categorization from Schnabel et al 2015)
Extrinsic Evaluation
How Do I Choose Embeddings?
When are Pre-trained Embeddings Useful?
Limitations of Embeddings
Unsupervised Coordination of Embeddings
Retrofitting of Embeddings to Existing Lexicons . We have an existing lexicon like WordNet, and would like our vectors to match (Faruqui et al. 2015)
Sparse Embeddings
De-biasing Word
FastText Toolkit