YoVDO

Neural Nets for NLP - Efficiency Tricks for Neural Nets

Offered By: Graham Neubig via YouTube

Tags

Neural Networks Courses Natural Language Processing (NLP) Courses Importance Sampling Courses

Course Description

Overview

Learn efficiency tricks for neural networks in natural language processing through this lecture from CMU's Neural Networks for NLP course. Explore GPU training tips, parallel training techniques, negative sampling methods, and softmax approximations like hierarchical softmax. Discover practical strategies to reduce operations, optimize CPU-GPU data movement, and manage memory effectively. Dive into various parallelism types, tackle computation challenges with large vocabularies, and examine advanced techniques like importance sampling, noise contrastive estimation, and efficient maximum inner product search. Gain insights into structure-based approximations, including class-based softmax and binary code prediction, to enhance neural network performance in NLP tasks.

Syllabus

Intro
Why are Neural Networks Slow and What Can we Do?
A Simple Example • How long does a metro-matrix multiply take?
Practically
Speed Trick 3
Reduce # of Operations
Reduce CPU-GPU Data Movement
What About Memory?
Three Types of Parallelism
Within-operation Parallelism
Operation-wise Parallelism
Example-wise Parallelism
Computation Across Large Vocabularies
A Visual Example of the Softmax
Importance Sampling (Bengio and Senecal 2003)
Noise Contrastive Estimation (Mnih & Teh 2012)
Mini-batch Based Negative Sampling
Hard Negative Mining • Select the top n hardest examples
Efficient Maximum Inner Product Search
Structure-based Approximations
Class-based Softmax (Goodman 2001) • Assign each word to a class
Binary Code Prediction (Dietterich and Bakiri 1995, Oda et al. 2017)
Two Improvement to Binary Code Prediction


Taught by

Graham Neubig

Related Courses

Natural Language Processing
Columbia University via Coursera
Natural Language Processing
Stanford University via Coursera
Introduction to Natural Language Processing
University of Michigan via Coursera
moocTLH: Nuevos retos en las tecnologías del lenguaje humano
Universidad de Alicante via Miríadax
Natural Language Processing
Indian Institute of Technology, Kharagpur via Swayam