SOURCE-LINKED INTELLIGENCE

Distance generalization in transformers: why bother with positional encoding?

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. Here we focus on distance generalization, which probes performance when inter-token distances are changed between training and inference, while keeping a fixed context length. We construct two synthetic delay copy tasks, both involving finite distances between source and recall, where tokens are copied either fully or selectively, and test models on delays unseen during training. We address three questions: (A) Do positional encoding schemes such

Read original source ↗ Open in workspace

recordType

paper

region

Global