Secure Self-supervised Learning: Challenges and Solutions

Explore secure self-supervised learning in this Google TechTalk presented by Neil Gong as part of the Differential Privacy for ML series. Delve into the challenges of supervised learning and discover the potential of self-supervised learning techniques. Learn about data augmentation and pre-training encoders using methods like SimCLR. Examine backdoor attacks, their effectiveness, and strategies to quantify their impact. Investigate existing defenses and their limitations. Gain insights into data auditing techniques, including membership inference-based approaches for pre-trained encoders. Understand the process of shadow training and building inference classifiers. Evaluate the concepts presented through real-world examples and experimental setups, including an assessment of CLIP. Enhance your understanding of secure machine learning practices and their implications for privacy and data protection.

Intro
Conventional Paradigm: Supervised Learning
Key Challenge of Supervised Learning
Road Map
Background on Self-supervised Learning
Data Augmentation
Pre-training an Encoder - SimCLR [ICML'20]
Building a Downstream Classifier
Backdoor Attack
Key Idea of Our Attack
Quantifying Effectiveness Goal
Quantifying Condition I
Quantifying Utility Goal
Optimization Problem
Attack Setting
Attack Success Rate
Clean Accuracy and Backdoored Accuracy
Evaluation on Real-world Pre-trained Encoders
Existing Defenses are Insufficient
Summary
Motivation on Data Auditing
Auditing Unauthorized Data Use
Examples of Real-world Unauthorized Data Use
Our EncoderMI: Membership Inference based Data Auditing for Pre-trained Encoders
Revisiting Encoder Pre-training
Shadow Training Setup
Pre-training a Shadow Encoder
Constructing a Training Set for Inference Classifier
Building an Inference Classifier
Experimental Setup
Evaluation on CLIP
Conclusion