SOURCE-LINKED INTELLIGENCE

On-Demand Attention: Language Models Know When to Recall

arXiv · AI, language, vision and robotics · article · Sep 17, 2026 · UTC

Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. We show that a pretrained model's decoding states already contain information predictive of this benefit, before the global read. Building on this finding, we introduce On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation. ODA trains only the recall head, leaving p

Read original source ↗ Open in workspace

recordType

paper

region

Global