SOURCE-LINKED INTELLIGENCE

Learning Length-Extrapolatable Recurrent Models

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state

Read original source ↗ Open in workspace

recordType

paper

region

Global