Learning Representations Using Causal Invariance - Leon Bottou

Explore a comprehensive lecture on learning representations using causal invariance delivered by Leon Bottou from Facebook AI Research at the Institute for Advanced Study's Workshop on Theory of Deep Learning. Delve into the challenges of machine learning, examining statistical problems as proxies and the impact of spurious correlations. Investigate the concept of multiple environments, the importance of negative mixtures, and the power of invariance in extrapolation. Analyze the relationship between invariance and causation, exploring applications in causal inference and adversarial domain adaptation. Examine robust supervised learning techniques, including linear least squares and nonlinear versions, and discover how these concepts apply to real-world scenarios such as colored MNIST datasets. Gain insights into scaling up invariant regularization and interpreting the underlying phenomena in this thought-provoking 33-minute presentation.

Intro
Joint work with
Summary
Why machine learning?
The statistical problem is only a proxy Example: detection of the action giving a phone call
A conjecture about adversarial features
Spurious correlations
Past observations
Nature does not shuffle the data. We do!
Multiple environments
Negative mixtures matter! Consider a search engine query classification problem
Learning stable properties
Invariance buys extrapolation powers
Trivial existence cases
Playing with the function family
Invariant representation
Finding the relevant variables
Invariance and causation
Invariance for causal inference
Invariant causal prediction
Adversarial Domain Adaptation
4- Robust supervised learning
The linear least square case
Issues
Characterization of the solutions
Rank of the feature matrix S
Exact recovery of high rank solutions Two set of environments
Nonlinear version
Colored MNIST
Scaling up invariant regularization
Phenomenon and interpretation