SOURCE-LINKED INTELLIGENCE

BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Long-context reasoning for large language models (LLMs) is becoming increasingly important, but training over long sequences remains challenging due to massive memory and communication requirements. Sequence parallelism has emerged as an essential technique for addressing bottlenecks in long sequence LLM training. However, we observe that existing sequence parallelism methods are batch-agnostic and apply uniform sequence partitioning across all batch sizes, resulting in inefficient communication. In this paper, we introduce Batch- Aware Sequence Parallelism (BASP), a sequence parallelism appro

Read original source ↗ Open in workspace

recordType

paper

region

Global