YoVDO

Building Makemore - Activations & Gradients, BatchNorm

Offered By: Andrej Karpathy via YouTube

Tags

Natural Language Processing (NLP) Courses Deep Learning Courses Neural Networks Courses Batch Normalization Courses Multilayer Perceptrons Courses

Course Description

Overview

Dive deep into the internals of multi-layer perceptrons (MLPs) in this comprehensive video lecture. Explore the statistics of forward pass activations and backward pass gradients, while learning about potential pitfalls in improperly scaled networks. Discover essential diagnostic tools and visualizations for assessing the health of deep networks. Understand the challenges of training deep neural networks and learn about Batch Normalization, a key innovation that simplifies the process. Gain practical insights through code examples, real-world applications, and visualizations. Complete provided exercises to reinforce your understanding of weight initialization and BatchNorm implementation. Follow along as the lecture covers topics such as Kaiming initialization, PyTorch implementation, and various visualization techniques for network analysis.

Syllabus

intro
starter code
fixing the initial loss
fixing the saturated tanh
calculating the init scale: “Kaiming init”
batch normalization
batch normalization: summary
real example: resnet50 walkthrough
summary of the lecture
just kidding: part2: PyTorch-ifying the code
viz #1: forward pass activations statistics
viz #2: backward pass gradient statistics
the fully linear case of no non-linearities
viz #3: parameter activation and gradient statistics
viz #4: update:data ratio over time
bringing back batchnorm, looking at the visualizations
summary of the lecture for real this time


Taught by

Andrej Karpathy

Related Courses

Neural Networks for Machine Learning
University of Toronto via Coursera
Good Brain, Bad Brain: Basics
University of Birmingham via FutureLearn
Statistical Learning with R
Stanford University via edX
Machine Learning 1—Supervised Learning
Brown University via Udacity
Fundamentals of Neuroscience, Part 2: Neurons and Networks
Harvard University via edX