Predictive Maps in the Brain

Explore a comprehensive theory of reinforcement learning that bridges the gap between "model-based" and "model-free" approaches in this 53-minute lecture by Sam Gershman from Harvard University. Delve into the concept of "predictive maps" in the brain and their role in efficiently computing values. Discover how this theory explains various aspects of hippocampal spatial representation and how its eigendecomposition reveals latent structures resembling entorhinal grid cells. Examine evidence from a novel revaluation task demonstrating how humans utilize predictive maps in reinforcement learning tasks. Investigate the role of dopamine error signals in learning these predictive maps. Cover topics such as cognitive maps, path integration, sequential decision problems, successor representation, grid cells, and hierarchical reinforcement learning. Gain insights into the relationship between grid cells and place cells, as well as evidence for population coding in the brain.

Intro
Outline
Origins of the cognitive map
What exactly is the cognitive map?
Path integration (dead reckoning)
Problems with the classical definition
From navigation to reinforcement learning
Sequential decision problems
Evidence for two learning systems
Cognitive map = model-based RL?
Cognitive map = predictive code?
Representing the environment
Encode Euclidean distance
Encode predictive statistics
Successor Representation
Asymmetric direction selectivity
Constraint by barriers
Context preexposure facilitation
Entorhinal grid cells
Grid cells via eigendecomposition
Dorsal-ventral axis
Eigenvector Grid Fields
Compartmentalization
Relationship between grid cells and place cells
Grid cells as a regularization network
Supporting evidence
Spatial structure is useful
Hierarchical reinforcement learning
Task design
Model predictions
How is the SR learned?
Evidence for population coding