SOURCE-LINKED INTELLIGENCE

Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-$k$: reducing $k$ at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top $k_1$ experts while normalizing by the probability mass of the top $k_2$ experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing fro

Read original source ↗ Open in workspace

recordType

paper

region

Global