Premonitions of Public Data for Private ML

Explore the intersection of private machine learning and public data in this 29-minute conference talk by Gautam Kamath from the University of Waterloo. Delve into the concept of differentially private stochastic gradient descent (DPSGD) and its application to datasets like MNIST and CIFAR-10. Examine the impact of public pre-training on private machine learning models, including zero-shot learning techniques for ImageNet. Consider the ethical implications of using public data in private ML contexts, addressing concerns about personal information and privacy. Gain insights into the future of private machine learning and the importance of developing meaningful benchmarks for privacy in the field.

Intro
Private Machine Learning
Stochastic Gradient Descent (SGD)
Differentially Private Stochastic Gradient Descent (DPSGD)
MNIST
The story so far
Enter "pre-training"
Pre-training for privacy
Why is (public) pre-training useful?
Does public pre-training help? (CIFAR-10)
Does public pre-training help? ImageNet
Zero-Shot Learning for ImageNet
JFT what?
Zero-Shot Learning is taking over
Is this really "private"?
Your secrets
Personal Information
Secrets about you
How can private ML stay relevant?
Focus on more meaningful benchmarks fe privacy
Closing