Train Short, Test Long - Attention With Linear Biases Enables Input Length Extrapolation

Explore the innovative ALiBi (Attention with Linear Biases) method for improving sequence extrapolation in transformer models. Dive into the limitations of traditional position encodings and discover how ALiBi's simple yet effective approach allows for efficient extrapolation to longer sequences than seen during training. Learn about the implementation details, including how to choose the slope parameter, and examine experimental results demonstrating ALiBi's performance advantages. Gain insights into why this method leads to better outcomes and understand its potential impact on natural language processing tasks.

- Intro & Overview
- Position Encodings in Transformers
- Sinusoidial Position Encodings
- ALiBi Position Encodings
- How to choose the slope parameter
- Experimental Results
- Comments & Conclusion