SOURCE-LINKED INTELLIGENCE

Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing

arXiv · AI, language, vision and robotics · article · Sep 15, 2026 · UTC

Sparse mixture-of-experts models route each token through a sequence of expert selections. We ask whether the immediately preceding selection adequately summarizes this trajectory for predicting the next router. Using frozen OLMoE and JetMoE models, we measure the held-out predictive gain from earlier expert selections while retaining the most recent selection as a common baseline. In OLMoE, extending the history from one to eleven layers raises router-logit $R^2$ from 0.59879 to 0.66544. A preregistered JetMoE replication yields four-layer gains of 0.14275 and 0.20528 at two target depths, wi

Read original source ↗ Open in workspace

recordType

paper

region

Global