What Matters in On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Explore a comprehensive analysis of on-policy reinforcement learning in this 38-minute video. Delve into the impact of various design choices on agent performance across five continuous control environments. Learn about parameterized agents, unified online RL frameworks, policy loss, network architectures, and initial policy considerations. Examine the effects of normalization, clipping, advantage estimation, and training setup on RL outcomes. Investigate timestep handling, optimizer selection, and regularization techniques. Gain valuable insights and practical recommendations for implementing effective on-policy RL agents based on extensive empirical research involving over 250,000 trained agents.

- Intro & Overview
- Parameterized Agents
- Unified Online RL and Parameter Choices
- Policy Loss
- Network Architecture
- Initial Policy
- Normalization & Clipping
- Advantage Estimation
- Training Setup
- Timestep Handling
- Optimizers
- Regularization
- Conclusion & Comments