Neural Nets for NLP - Multi-task, Multi-lingual Learning

Explore multi-task and multi-lingual learning in natural language processing through this comprehensive lecture from CMU's Neural Networks for NLP course. Delve into key concepts like feature extraction, increasing data through multi-tasking, pre-training encoders, and regularization techniques. Examine various approaches to multi-task learning, including selective parameter adaptation, soft parameter tying, and supervised domain adaptation. Investigate multilingual structured prediction, sequence-to-sequence models, and the challenges of handling multiple annotation standards. Gain insights into designing effective multi-task and multi-lingual NLP systems through practical examples and design principles presented by Antonis Anastasopoulos.

Intro
Remember, Neural Nets are Feature Extractors!
Types of Learning
Plethora of Tasks in NLP
Rule of Thumb 1: Multitask to Increase Data
Rule of Thumb 2
Standard Multi-task Learning
Examples of Pre-training Encoders . Common to pre-train encoders for downstream tasks, common to use
Regularization for Pre-training (e.g. Barone et al. 2017) Pre-training relies on the fact that we won't move too far from the
Selective Parameter Adaptation Sometimes it is better to adapt only some of the parameters
Soft Parameter Tying
Supervised Domain Adaptation through Feature Augmentation
Unsupervised Learning through Feature Matching
Multilingual Structured Prediction/ Multilingual Outputs • Things are harder when predicting a sequence of actions (parsing) or words (MT) in different languages
Multi-lingual Sequence-to- sequence Models
Types of Multi-tasking
Multiple Annotation Standards
Different Layers for Different
Summary of design dimensions