SOURCE-LINKED INTELLIGENCE

Higher-Dimensional Rotary Position Embedding

arXiv · AI, language, vision and robotics · article · Aug 30, 2026 · UTC

Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhanc

Read original source ↗ Open in workspace

recordType

paper

region

Global