Not All Memories Are Created Equal - Learning to Forget by Expiring

Explore a detailed explanation of Facebook AI's Expire-Span, a Transformer XL variant that dynamically assigns expiration dates to previously encountered signals. Learn how this innovative approach allows for processing sequences of thousands of tokens while maintaining manageable memory and compute requirements. Discover the architecture, advantages, and potential limitations of Expire-Span, as well as its performance compared to baseline systems. Gain insights into remembering past information in sequence models, learning to expire memories, and the differences between Expire-Span and local attention. Examine the experimental results and understand how this method can improve efficiency in long-context tasks such as character-level language modeling and frame-by-frame moving object tracking.

- Intro & Overview
- Remembering the past in sequence models
- Learning to expire past memories
- Difference to local attention
- Architecture overview
- Comparison to Transformer XL
- Predicting expiration masks
- Experimental Results
- Conclusion & Comments