SOURCE-LINKED INTELLIGENCE

Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length generalization benchmarks. We present a provably correct, transformer parameterization (with only 280 learnable parameters for Boolean algebra tasks) capable of learning and evaluating problems of any depth or length. We assume inputs are fully parenthesized, well-formed expressions. Our approach conceptualizes algorithmic tasks as circuit models embedded in transformers, enabling depth-1 circuit reduction in a single forward pass. To achieve

Read original source ↗ Open in workspace

recordType

paper

region

Global