SOURCE-LINKED INTELLIGENCE

Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

Block diffusion language models (BDLMs) combine autoregressive dependencies across blocks with parallel denoising within blocks, but long-context training is constrained by distributed attention communication and activation memory. Conventional context parallelism (CP) shards the combined clean-plus-corrupted sequence by position, communicating shared clean K/V together with block-specific corrupted K/V and their gradients. We observe that the BDLM objective separates over target blocks. We introduce block parallelism (BP), a new distributed parallelism dimension that assigns each corrupted-bl

Read original source ↗ Open in workspace

recordType

paper

region

Global