SOURCE-LINKED INTELLIGENCE

Higher-order pruning of experts in mixture-of-experts language models

arXiv · Artificial Intelligence · article · Sep 16, 2026 · UTC

Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts' contributions are purely additive. In reality, expert usage in MoEs is inherently cooperative. We derive HOPE (Higher-Order Pruning of Experts), a second-order pruning objective which provably minimizes an upper bound on the error resulting from pruning. We show that REAP (a state-of-the-art first-order

Read original source ↗ Open in workspace

recordType

paper

region

Global