SOURCE-LINKED INTELLIGENCE

Length Generalization for Transformers via Compression

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Recent advancements in transformer length generalization theory enable us to reliably predict when a transformer can learn to solve a task. In particular, the C-RASP hypothesis (a formalized version of the so-called RASP-l conjecture) posits that transformers length-generalize on a task if and only if a solution is expressible in the C-RASP language. While this hypothesis has strong empirical validation, theoretical problems arise from the fact that no computable length generalization bounds exist for C-RASP, alongside the discovery of seemingly contradictory experiments. To address these prob

Read original source ↗ Open in workspace

recordType

paper

region

Global