A Critical Analysis of Self-Supervision, or What We Can Learn From a Single Image

Explore a critical analysis of self-supervision in deep learning through this informative video lecture. Delve into the intriguing question of whether self-supervision truly requires vast amounts of data, and discover how a single image can be sufficient to train the lower layers of a deep neural network. Learn about the paper's methodology, including the use of linear probes, and examine the surprising results that challenge conventional wisdom. Gain insights into popular self-supervision techniques such as BiGAN, RotNet, and DeepCluster, and understand their effectiveness when applied to limited data sets. Investigate the role of data augmentation in achieving comparable results to those obtained with millions of images and manual labels. Analyze the implications of these findings for the field of deep learning, particularly in understanding the information content of early network layers and the potential for synthetic transformations to capture low-level image statistics.

- Overview
- What is self-supervision
- What does this paper do
- Linear probes
- Linear probe results
- Results
- Learned Features