SOURCE-LINKED INTELLIGENCE

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific ch

Read original source ↗ Open in workspace

recordType

paper

region

Global