Neural Nets for NLP - Minimum Risk Training and Reinforcement Learning

Explore minimum risk training and reinforcement learning in natural language processing through this comprehensive lecture from CMU's Neural Networks for NLP course. Delve into concepts such as error and risk minimization, policy gradient methods, REINFORCE algorithm, and value-based reinforcement learning. Learn about techniques for stabilizing reinforcement learning, including adding baselines and increasing batch sizes. Understand the applications and challenges of reinforcement learning in NLP tasks, and gain insights into when to use these approaches effectively. Discover methods for estimating value functions and addressing problems like exposure bias and disregard for evaluation metrics in neural language models.

Intro
Problem 1: Exposure Bias
Problem 2: Disregard to Evaluation Metrics
Error
Problem: Argmax is Non- differentiable
Sampling for Risk
Adding Temperature
What is Reinforcement Learning?
Why Reinforcement Learning in NLP?
Supervised MLE
Self Training
Policy Gradient/REINFORCE
Credit Assignment for Rewards
Problems w/ Reinforcement Learning
Adding a Baseline
Calculating Baselines
Increasing Batch Size
Warm-start
When to Use Reinforcement Learning?
Action-Value Function
Estimating Value Functions