SOURCE-LINKED INTELLIGENCE

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

Mixture-of-experts (MoE) inference on consumer hardware is bounded by weight memory: a 35B-class model is 19.5GB at 4-bit, and sparsity shrinks the compute per token, not the bytes that must be held. Naive offloading to SSD does not help on its own, because layer N+1's experts must be chosen before layer N's output exists, so the reads cannot start early enough to hide behind compute. We present Edge0, a streaming MoE inference engine that closes the gap with a prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself, s

Read original source ↗ Open in workspace

recordType

paper

region

Global