SOURCE-LINKED INTELLIGENCE

Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additio

Read original source ↗ Open in workspace

recordType

paper

region

Global