SOURCE-LINKED INTELLIGENCE

Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Long context language models now advertise windows of one million tokens, but two habits limit how much of that window is used. Attention heads with nothing useful to read still spend their budget on the first token, which is called the attention sink, and where a fact sits in the context changes whether the model finds it. Gated attention cut first token attention from 46.7 percent to 4.8 percent at NeurIPS 2025, and Kimi K3 pairs that idea with Kimi Delta Attention and Attention Residuals behind a one million token window, eight times past the range where these diagnostics have been reported

Read original source ↗ Open in workspace

recordType

paper

region

Global