SOURCE-LINKED INTELLIGENCE

Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Mixture-of-experts (MoE) architectures scale large language models efficiently, but they demand massive GPU memory. To cope with such demand, models are commonly compressed to reduce their memory footprint. Residual sparsification is a representative compression technique that decomposes each projection matrix of an expert into a shared base matrix and per-expert residual matrix, and then compresses the residuals. Existing sparsification methods compress each residual matrix independently by minimizing its compression error, thereby minimizing the error of each projection matrix. However, our

Read original source ↗ Open in workspace

recordType

paper

region

Global