Neural Nets for NLP 2019: Sentence and Contextualized Word Representations

Explore sentence and contextualized word representations in this comprehensive lecture from CMU's Neural Networks for NLP course. Dive into sentence representations, contextual word representations, and their applications in various NLP tasks. Learn about sentence classification, paraphrase identification, and textual entailment. Discover different types of learning, multi-task learning, and pre-trained representations. Examine language model transfer, context prediction transfer, and entailment transfer. Gain insights into bi-directional language modeling objectives and masked word prediction techniques. Enhance your understanding of advanced NLP concepts and their practical implementations in this in-depth presentation by Graham Neubig.

Intro
Goal for Today
Where would we need/use Sentence Representations?
Sentence Classification
Paraphrase Identification (Dolan and Brockett 2005) • Identify whether A and B mean the same thing
Textual Entailment (Dagan et al. 2006, Marelli et al. 2014)
Model for Sentence Pair Processing
Types of Learning
Plethora of Tasks in NLP
Rule of Thumb 2
Standard Multi-task Learning
Thinking about Multi-tasking, and Pre-trained Representations
General Model Overview
Language Model Transfer
End-to-end vs. Pre-training
Context Prediction Transfer (Skip-thought Vectors) (Kiros et al. 2015)
Paraphrase ID Transfer (Wieting et al. 2015)
Large Scale Paraphrase Data (ParaNMT-50MT) (Wieting and Gimpel 2018)
Entailment Transfer (InferSent) (Conneau et al. 2017)
Bi-directional Language Modeling Objective (ELMO)
Masked Word Prediction (BERT)