Example Memorization in Learning: Batch and Streaming - Differential Privacy for ML

Explore the concept of example memorization in learning through this Google TechTalk presented by Gavin Brown as part of the Differential Privacy for ML seminar series. Delve into the meaning of "memorizing training examples" and examine empirical example memorization. Investigate the relationships between space, information, and deep learning, with a focus on Shannon's mutual information. Learn about theorems related to memorizing entire examples and tasks involving mixtures of subpopulations and per-subpopulation distributions. Discover proof techniques for lower bounds via singletons and examine experiments involving logistic regression and neural networks. Analyze the setup for learning from a stream of examples and explore theorems on space requirements and example memorization. Investigate space lower bounds for natural models and understand the structure and overview of proofs, including requirements for distinguishing one bit. Gain insights into the main theorems and their implications, and explore directions for future work in the field of example memorization in learning.

Intro
What do we mean? "Memorizing training examples"
Empirical Example Memorization
Space, Information, and Deep Learning
Important preliminary: Shannon's mutual information
Theorem: Memorizing entire examples
Tasks: Mixtures of subpopulations
Tasks: Per-subpopulation distributions
Proof: Lower bounds via singletons
Experiments: Logistic regression and neural network
Setup: Learning from a stream of examples
Theorem: How Much Space?
Theorem: Example Memorization
Tasks: Space Lower Bounds for Natural Models
Proof: Structure and Overview
Proof: Requirements for distinguishing one bit
Main theorems and implications
Directions for future work
Memorize when you can't identify relevant information