Does Learning Require Memorization? A Short Tale About a Long Tail

Explore the relationship between learning and memorization in machine learning through a thought-provoking 25-minute ACM conference talk. Delve into concepts such as overfitting, label memorization, and the role of theory in learning. Examine the importance of interpolation, hard atypical examples, and subpopulations in dataset analysis. Investigate the challenges posed by long-tailed data distributions and their impact on model performance. Discuss the benefits and potential drawbacks of fitting data, including its connection to memorization. Extend the analysis beyond discrete domains and explore the concept of coupling. Review experimental validation conducted with Chiyuan Zhang, and draw insightful conclusions about the nature of learning and memorization in artificial intelligence.

Intro
(Over?)fitting the dataset
Label memorization
What about theory?
Interpolation
Hard atypical examples
Subpopulations
Long-tailed data
The tail rears its head
Model
Benefits of fitting
Fitting and memorization
Beyond discrete domains
Coupling
Experimental validation (with Chiyuan Zhang)
Conclusions