SOURCE-LINKED INTELLIGENCE

Content-Based Addressing for Long Context

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local token order, but increasing context length creates a positional train-test mismatch: RoPE produces relative rotations at offsets not seen during training. Methods that rescale, interpolate, randomize, or bias positions specify how attention handles those offsets, but still derive positional information from a growing token counter. We instead divide a token stream into units, retain ordinary RoPE positions within each unit, and assign every completed

Read original source ↗ Open in workspace

recordType

paper

region

Global