What Do Our Models Learn? - Aleksander Mądry

Explore the intricacies of machine learning models and their learning processes in this comprehensive lecture by Aleksander Mądry at the Institute for Advanced Study. Delve into the ML research pipeline, examining concerns such as classic and adaptive overfitting. Investigate the role of background bias in image classification, including studies on ImageNet-9 and adversarial backgrounds. Analyze the creation and validation of datasets, focusing on crowdsourced methods and the challenges of multi-object images. Evaluate the impact of dataset replication on model performance, using ImageNet-v2 as a case study. Gain insights into human-based evaluation techniques and the potential for statistical bias in machine learning research.

Intro
ML Research Pipeline
Concern #1: "Classic" Overfitting
Concern #2: Adaptive Overfitting
Simple Setting: Background bias
Do Backgrounds Contain Signal?
ImageNet-9: A Fine-Grained Study Xiao Engstrom Ilyas M 2020
Adversarial Backgrounds
Background-Robust Models?
Are Better Models Better?
Biases Can Be Subtle
How Are Datasets Created?
Dataset Creation in Practice
Crowdsourced Validation: A Closer Look
Prerequisite: Detailed Annotations
Restricting Relevant Labels
From Validation to Classification
Multi-Object Images
How Does This Affect Accuracy?
Which Object Do Models Predict?
Human-Based Evaluation
Dataset Replication
Case Study: ImageNet-v2
Replication Pipeline
Statistical Bias