dLoRA - Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving

Explore a cutting-edge conference talk on dLoRA, an innovative inference serving system for LoRA (Low-Rank Adaptation) models in large language model (LLM) serving. Delve into the dynamic orchestration of requests and LoRA adapters, focusing on two key aspects: dynamically merging and unmerging adapters with the base model, and migrating requests and adapters between worker replicas. Discover the insights behind these capabilities, including the impact of request skewness on adapter merging decisions and the load imbalance caused by varying input and output lengths in autoregressive LLM requests. Learn about the credit-based batching algorithm for merge/unmerge decisions and the request-adapter co-migration algorithm. Examine the impressive performance improvements achieved by dLoRA, with throughput increases of up to 57.9× and 26.0× compared to vLLM and HugginFace PEFT, respectively, and up to 1.8× lower average latency than the concurrent work S-LoRA.

OSDI '24 - dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving