SOURCE-LINKED INTELLIGENCE

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

arXiv · AI, language, vision and robotics · article · Sep 15, 2026 · UTC

We introduce a simple architectural modification to decoder-only transformers: a persistent recurrent state that observes hidden representations via cross-attention, updates itself through a GRU, and modulates subsequent processing via gated addition. Inserted between the lower and upper halves of a 6-layer transformer, this module adds only 3.7\% additional parameters while reducing evaluation loss from $2.438 \pm 0.004$ to $1.743 \pm 0.018$, corresponding to a 28.5\% reduction on held-out language modeling data. The improvement is statistically significant across 5 random seeds ($p < 0.01$)

Read original source ↗ Open in workspace

recordType

paper

region

Global