Fairness in Serving Large Language Models

Explore a 16-minute conference talk from USENIX's OSDI '24 program that delves into the challenges of ensuring fairness in serving Large Language Models (LLMs). Learn about the novel Virtual Token Counter (VTC) scheduling algorithm designed to address the unique challenges posed by LLM inference services. Discover how this approach improves upon traditional request rate limits by accounting for input and output tokens processed, leading to better resource utilization and client experience. Examine the proof of a 2× tight upper bound on service differences between backlogged clients and understand how VTC outperforms baseline methods in various conditions. Gain insights into the complexities of fair scheduling for LLMs, considering their unpredictable request lengths and batching characteristics on parallel accelerators. Access the reproducible code and dive deeper into this cutting-edge research on fairness in LLM serving.

OSDI '24 - Fairness in Serving Large Language Models