Neural Nets for NLP - Latent Variable Models

Explore latent variable models in neural networks for natural language processing in this comprehensive lecture from CMU's Neural Networks for NLP course. Delve into the differences between generative and discriminative models, as well as deterministic and random variables. Learn about Variational Autoencoders (VAEs) and their applications in NLP, including techniques for handling discrete latent variables. Discover the challenges in training latent variable models and strategies to overcome them, such as KL divergence annealing and weakening the decoder. Examine methods for dealing with discrete latent variables, including enumeration, sampling, and reparameterization techniques. Access accompanying slides and code examples to reinforce your understanding of these advanced concepts in neural network-based NLP.

Intro
Discriminative vs. Generative Models
Quiz: What Types of Variables?
Why Latent Random Variables?
A Latent Variable Model
What is Our Loss Function? . We would like to maximize the corpus log likelihood
Disconnect Between Samples and Objective
VAE Objective . We can create an optimizable objective matching our problem, starting with KL divergence
Interpreting the VAE Objective
Problem: Straightforward Sampling is Inefficient Current
Problem! Sampling Breaks Backprop
Solution: Re-parameterization Trick
Generating from Language Models
Motivation for Latent Variables
Difficulties in Training
KL Divergence Annealing
Weaken the Decoder
Discrete Latent Variables?
Enumeration
Method 2: Sampling
Reparameterization (Maddison et al. 2017. Jang et al. 2017)