Robust Deep Learning Under Distribution Shift

Explore the challenges and solutions in deep learning under distribution shift in this 50-minute lecture by Zack Lipton from Carnegie Mellon University. Delve into topics such as adversarial misspellings, feedback loops, and label shift detection. Learn about black box shift estimation and correction techniques, and examine their applications in pneumonia prediction and image classification. Investigate the impact of implicit bias in SGD, weight-invariance, and regularization methods on deep learning models. Gain insights into domain adaptation strategies and synthetic experiments that address distribution shift problems in real-world scenarios.

Intro
Outline
Standard assumptions
Adversarial Misspellings (Char-Level Attack)
Curated Training Task Fail to Represent Reality
Feedback Loops
Impossibility absent assumptions
Detecting and correcting for label shift with black box predictors
Motivation 1: Pneumonia prediction
Epidemic
Motivation 2: Image Classification
The test-Item effect
Domain Adaptation - Formal Setup
Label Shift (aka Target Shift)
Contrast with Covariate Shift
Black Box Shift Estimation (BBSE)
Confusion matrices
Applying the label shift assumption...
Consistency
Error bound
Detection
Estimation error (MNIST)
Black Box Shift Correction (CIFAR10 w IW-ERM)
A General Pipeline for Detecting Shift
Non-adversarial image perturbations
Detecting adversarial examples
Covariate shift + model misspecification
Implicit bias of SGD on linear networks w. linearly separable data
Impact of IW on ERM decays over MLP training
Weight-Invariance after 1000 epochs
L2 Regularization v Dropout
Deep DA / Domain-Adversarial Nets
Synthetic experiments