SOURCE-LINKED INTELLIGENCE

RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States

arXiv · AI, language, vision and robotics · article · Sep 11, 2026 · UTC

Linear attention and state-space models provide linear-time sequence modeling, but their recurrent memory remains a second-order tensor (a matrix), limiting the order of interactions that can be represented in the state. We introduce the RunningTensor, which generalizes this memory to an order-$o$ tensor, updated by a rank-1 outer product and read by contracting against $o-1$ vector queries. Order $2$ recovers linear attention; we study order $3$ as a proof of concept, retaining both recurrent and parallel forms while remaining linear in sequence length $T$ and improving working memory capacit

Read original source ↗ Open in workspace

recordType

paper

region

Global