Towards Better Understanding of Contrastive Learning
Offered By: DataLearning@ICL via YouTube
Course Description
Overview
Explore a comprehensive presentation on contrastive learning delivered by Yuandong Tian from Meta for the Data Learning working group. Delve into the empirical success of deep models, self-supervised learning, and the formulation of contrastive learning. Examine the understanding of contrastive loss, the InfoNCE example, and coordinate-wise optimization. Discover a surprising connection to kernels and gain insights into nonlinear analysis. Investigate training dynamics, including 1-layer 1-node and multiple node nonlinear networks. Learn about conditional independence, global modulation, and feature emergence. Compare quadratic loss versus InfoNCE through experimental settings, model architecture, and evaluation metrics. This 58-minute talk, recorded on March 7, 2023, offers valuable insights for researchers and students developing new technologies based on Data Assimilation and Machine Learning.
Syllabus
Intro
Great Empirical Success of Deep Models
Self-supervised Learning (SSL)
Contrastive Learning (CL)
Formulation of Contrastive Learning
Understanding of Contrastive Loss
What Deep Learning Brings?
Example: InfoNCE
Coordinate-wise Optimization
A Surprising Connection to Kernels
Overview of Nonlinear Analysis
Nonlinear Setting
Training Dynamics
1-layer 1-node nonlinear network
How to reduce the local roughness p(w)?
1-layer multiple node nonlinear network
Assumptions
Conditional Independence
What linear network cannot do
Global modulation
Feature Emergence
Experiment Setting
Model Architecture & Evaluation Metric
Visualization
Quadratic Loss versus InfoNCE
Taught by
DataLearning@ICL
Related Courses
Stanford Seminar - Audio Research: Transformers for Applications in Audio, Speech and MusicStanford University via YouTube How to Represent Part-Whole Hierarchies in a Neural Network - Geoff Hinton's Paper Explained
Yannic Kilcher via YouTube OpenAI CLIP - Connecting Text and Images - Paper Explained
Aleksa Gordić - The AI Epiphany via YouTube Learning Compact Representation with Less Labeled Data from Sensors
tinyML via YouTube Human Activity Recognition - Learning with Less Labels and Privacy Preservation
University of Central Florida via YouTube