SOURCE-LINKED INTELLIGENCE

Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle each request. We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class. We develop the policy in a discrete event simulator and validate it on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. Al

Read original source ↗ Open in workspace

recordType

paper

region

Global