SOURCE-LINKED INTELLIGENCE

RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

arXiv · AI, language, vision and robotics · article · Sep 6, 2026 · UTC

Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention scales quadratically with sequence length and quickly becomes the dominant inference bottleneck. Sparse low-rank hybrids alleviate this cost by combining a local sparse branch with a global compressed branch. In video DiTs equipped with 3D Rotary Position Embeddings (RoPE), the global branch faces a structural compatibility issue: when RoPE is applied before a nonlinear feature map, the rotation and nonlinearity generally do not commute, making it difficult to keep a query-indep

Read original source ↗ Open in workspace

recordType

paper

region

Global