SOURCE-LINKED INTELLIGENCE

Informational Antilocality and the Locality Bias in LLMs

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of $k$ contiguous symbols. We construct such languages with increasing $k$, finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.

Read original source ↗ Open in workspace

recordType

paper

region

Global