Neural Nets for NLP - Efficiency Tricks for Neural Nets

Learn efficiency tricks for neural networks in natural language processing through this lecture from CMU's Neural Networks for NLP course. Explore GPU training tips, parallel training techniques, negative sampling methods, and softmax approximations like hierarchical softmax. Discover practical strategies to reduce operations, optimize CPU-GPU data movement, and manage memory effectively. Dive into various parallelism types, tackle computation challenges with large vocabularies, and examine advanced techniques like importance sampling, noise contrastive estimation, and efficient maximum inner product search. Gain insights into structure-based approximations, including class-based softmax and binary code prediction, to enhance neural network performance in NLP tasks.

Intro
Why are Neural Networks Slow and What Can we Do?
A Simple Example • How long does a metro-matrix multiply take?
Practically
Speed Trick 3
Reduce # of Operations
Reduce CPU-GPU Data Movement
What About Memory?
Three Types of Parallelism
Within-operation Parallelism
Operation-wise Parallelism
Example-wise Parallelism
Computation Across Large Vocabularies
A Visual Example of the Softmax
Importance Sampling (Bengio and Senecal 2003)
Noise Contrastive Estimation (Mnih & Teh 2012)
Mini-batch Based Negative Sampling
Hard Negative Mining • Select the top n hardest examples
Efficient Maximum Inner Product Search
Structure-based Approximations
Class-based Softmax (Goodman 2001) • Assign each word to a class
Binary Code Prediction (Dietterich and Bakiri 1995, Oda et al. 2017)
Two Improvement to Binary Code Prediction