SOURCE-LINKED INTELLIGENCE

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative toke

Read original source ↗ Open in workspace

recordType

paper

region

Global