Training on the Test Set and Other Heresies

Explore the controversial topic of training on test sets and other unconventional practices in machine learning through this thought-provoking lecture by Ben Recht from UC Berkeley. Delve into fundamental concepts of machine learning and generalization, challenging conventional wisdom in the field. Examine critical aspects such as regularization, random features, regression, boosting, and model size. Analyze the implications of diminishing returns, new holdout and test sets, and the variability in datasets like ImageNet. Gain insights into the complexities of data collection, including discussions on Mechanical Turk and Kaggle competitions. Question established norms and consider alternative approaches to improve machine learning practices and outcomes.

Intro
Conventional wisdom
What is machine learning
What is generalization
What can we take away
Least favorite figure
Inception model
Regularization
Pull Request
Random Features
Regression
Boosting
Model Size
Diminishing Returns
New Holdout Set
New Test Set
Results
Mechanical Turk
Variability
Imagenet Data
Cotton Pickers
Kaggle