Building Embodied Autonomous Agents

Explore the cutting-edge developments in embodied autonomous agents through this comprehensive seminar by Ruslan Salakhutdinov at MIT. Delve into modular agent design for visual navigation, natural language instruction following, efficient exploration, and long-term planning. Learn about the innovative Self-supervised Embodied Active Learning (SEAL) framework, which utilizes 3D semantic maps to enhance both action and perception in a self-supervised manner. Discover how SEAL improves object detection and instance segmentation while boosting performance in object goal navigation tasks. Examine a novel embodied instruction following method that employs structured representations and semantic search policies to achieve state-of-the-art performance in the ALFRED environment. Gain insights into the benefits of explicit spatial memory and semantic search policies for more robust and generalizable state-tracking and guidance. Throughout the seminar, explore topics such as physical intelligence, goal-conditioned navigation, active neural SLAM, domain generalization, topological maps, and the transition from simulation to real-world applications in building intelligent agents.

Intro
Learning Behaviors
Physical Intelligence
Goal-conditioned Navigation
Real World: Object Goal Navigation
Navigation Tasks
Active Neural SLAM: Overview
Neural SLAM Module
Domain Generalization: Matterport3D
Exploration Results
Point-Goal Navigation
Harder Datasets
Semantic Priors and Common Sense
Topological Maps
Topological Graph Representation
Semantic Prediction
Neural Topological SLAM
Internet vs Embodied Data
Using Internet models for Embodied Agents
Embodied Perception
Perception-Action Loop
SEAL: Self-supervised Embodied Active Learning
3D Semantic Mapping
Gainful Curiosity
Policy Learning
3D Label Propagation
Explicit Semantic Mapping
Results: Object Goal Navigation
EIF: Embodied Instruction Following: ALFRED
Simulation to Real
Building Intelligent Agents