Sequence Modeling with Neural Networks

Explore sequence modeling with neural networks in this lecture from MIT's Introduction to Deep Learning course. Delve into the challenges of modeling sequential data, understand the limitations of fixed window approaches, and discover how Recurrent Neural Networks (RNNs) address these issues. Learn about backpropagation through time, the vanishing gradient problem, and solutions like gated cells. Examine practical applications such as music generation and machine translation, and understand advanced concepts like attention mechanisms. Gain insights into activation functions, initialization techniques, and the importance of parameter sharing in sequence modeling.

Intro
What is a sequence?
a sequence modeling problem
idea: use a fixed window
problem: we can't model long-term dependencies
idea: use entire sequence, as a set of counts
idea: use a really big fixed window
problem: no parameter sharing
to model sequences, we need
example network
RNNS remember their previous state
"unfolding" the RNN across time
remember: backpropagation
let's try it out for W with the chain rule
backpropagation through time
problem: vanishing gradient
activation functions
initialization
gated cells
possible task: music generation
possible task: machine translation
problem: a single encoding is limiting
solution: attend over all encoder states