SOURCE-LINKED INTELLIGENCE

SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linear attention state dynamics. In this work, we study long-context extension of Gated DeltaNet (GDN) from a spectral perspective of transition matrix and identify two essential factors governing long-range information retrieval: (1) a sufficiently broad slow spectral band aligned with the target depend

Read original source ↗ Open in workspace

recordType

paper

region

Global