SOURCE-LINKED INTELLIGENCE

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

arXiv · AI, language, vision and robotics · article · Sep 11, 2026 · UTC

Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods usually use a lightweight selector to score context units, followed by hard Top-K selection that blocks gradients from the language modeling loss. Consequently, these methods commonly distill layer-wise dense attention distributions. Although this encourages the selector to rank context units by dense attention weights in the original model, the ranking is not directly aligned wi

Read original source ↗ Open in workspace

recordType

paper

region

Global