Representation Learning of Grounded Language and Knowledge - With and Without End-to-End Learning

Explore representation learning of grounded language and knowledge in this lecture by Yejin Choi from the University of Washington. Delve into intelligent communication, types of knowledge, and end-to-end learning approaches. Examine practical applications in cooking instructions, biology wet lab procedures, and engine oil changes. Discover the unique challenges of procedural language and the use of action graphs. Investigate unsupervised learning techniques and the role of knowledge in models. Learn about recipe generation as a form of machine translation, including attention mechanisms and neural checklist models. Compare various baselines and examine neural recipe examples. Analyze the limitations of end-to-end learning and explore dynamic networks and verb physics frames. Conclude with insights on reverse engineering commonsense knowledge and the intersection of finite state automata with RNN language models.

Intro
Intelligent Communication
Types of Knowledge
End-to-End Learning
Cooking Instructions
Biology Wet Lab Instructions
How to Change Engine Oil
Unique Challenges of Procedural Language
Action graphs
Action graph for blueberry muffins
Unsupervised Learning
Knowledge in the Model
Learned cooking knowledge
How to generate recipes
Task Definition
Recipe generation as machine translation?
Encode title - decode recipe
Recipe generation vs machine translation
Let's make salsa!
Checklist is probabilistic
Hidden state classifier is soft
Interpolation
Choose ingredient via attention
Attention-generated embeddings
Baselines
Neural Recipe Example 1
Example: skillet chicken rice
Example: chocolate covered potato chips
Neural checklist model for dialogue generation
Hotel domain
What's missing in the end-to-end...
Dynamic ??? Networks
Representation: Verb Physics Frames
Reverse Engineering Commonsense Knowledge!
Conclusion (as of today)
Intersect FSA with RNN Language Model