SOURCE-LINKED INTELLIGENCE

LSTM-UT and Recurrent-Depth Transformers on Cellular Automata

arXiv · AI, language, vision and robotics · article · Sep 17, 2026 · UTC

Recurrent-depth Transformers apply shared computation repeatedly, but differ in how they retain information across steps. We compare a Block Universal Transformer (BUT), which carries only its current hidden state; CoTFormer, which also retains an expanding attention cache; and a new LSTM Universal Transformer (LSTM-UT) with bounded gated memory. On Rule 30 cellular automata, BUT extrapolates to unseen recurrent depths more reliably than CoTFormer, although its accuracy eventually degrades. State and cache interventions show that CoTFormer's failure depends on their interaction: correcting the

Read original source ↗ Open in workspace

recordType

paper

region

Global