SOURCE-LINKED INTELLIGENCE

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel δ-token fairness guarantee: for a well-behaved client,

Read original source ↗ Open in workspace

recordType

paper

region

Global