Generalization and Robustness in Offline Reinforcement Learning

Explore a comprehensive lecture on offline reinforcement learning, focusing on generalization and robustness. Delve into the challenges and potential applications of this learning paradigm, which uses pre-collected static datasets without further environment interaction. Examine a general model-based offline RL algorithm that demonstrates generalization in large-scale Markov Decision Processes and robustness in policy discovery. Investigate offline Imitation Learning, including an algorithm with polynomial sample complexity and state-of-the-art performance in continuous control robotics benchmarks. Cover topics such as empirical RL for large-scale problems, finite horizon MDPs, offline data collection and coverage, learning goals in offline RL, and traditional versus offline imitation learning.

Intro
Empirical RL for large-scale problems
Traditional Online RL paradigm
offline RL paradigm
Potential Applications of Offline RL
Why offline RL is challenging?
Finite Horizon MDPs
Offline Data Collection
Offline Data Coverage
Learning goal in Offline RL: Robustness
Learning goal in Offline RL: Generalization
A Model-based Approach
A Naive Model-based Approach
Formal Theoretical Guarantee for CPPO
Traditional Imitation Learning
Covariate shift in Imitation
Offline Imitation Learning
Explanation of MILO
Experiments of MILO