SparQ Attention: Bandwidth-Efficient LLM Inference

Explore a comprehensive presentation on SparQ Attention, delivered by Ivan Chelombiev and Luka Ribar from GraphCore. Delve into their groundbreaking work on increasing inference throughput of Large Language Models (LLMs) by reducing memory bandwidth requirements in attention blocks. Learn about the innovative technique of selective fetching of cached history, which can be applied to existing LLMs during inference without modifying pre-training or requiring additional fine-tuning. Discover how SparQ Attention can decrease attention memory bandwidth requirements up to eight times while maintaining accuracy, as demonstrated through evaluations of Llama 2 and Pythia models on various downstream tasks. Gain insights into the latest advancements in AI optimization and LLM efficiency, and understand the potential impact of this research on the future of language model deployment and performance.

We're very excited to welcome both Ivan Chelombiev and Luka Ribar from GraphCore. They will be presenting their work on SparQ Attention presentation starts at