SOURCE-LINKED INTELLIGENCE

Affinity-Aware Sharding for Delayed Tensor Parallelism

arXiv · AI, language, vision and robotics · article · Sep 12, 2026 · UTC

Delayed Tensor Parallelism (DTP) removes the blocking all-reduce of tensor-parallel Transformer inference. Every device adds its own partial output to its residual stream (and broadcasts it) immediately, but only gathers (receives) the other devices' partials $δ$ modules later. A TP to DTP change therefore amounts to a real architecture change, and dense Transformer models need to be retrained or distilled after adaptation. We show that DTP breaks the permutation symmetry of neurons inside FFNs and of KV heads inside attention modules, and that this symmetry breakage makes the sharding itself

Read original source ↗ Open in workspace

recordType

paper

region

Global