SOURCE-LINKED INTELLIGENCE

Hyperparameter Scaling Laws Across MoE Sparsity

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyperparameter transfer challenging. In this work, we show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs: the optimal learning rate and batch size vary with activation ratio, and these shifts cannot be explained by either total or activated parameter count alone. To characterize this dependence, we conduct 1,800 pre-training runs spanning six activated-parameter scales and models with up to 6B total non-embeddin

Read original source ↗ Open in workspace

recordType

paper

region

Global