SOURCE-LINKED INTELLIGENCE

When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study of how pruning affects SAE behavior and theoretically show that, for a fixed SAE, its impact is governed by perturbation energy, a covariance-weighted norm. This perspective exposes a key limitation of magnitude pruning: by ignoring activation geometry, it distorts the learned representation space and degrades SAE functionality. Activation-aware methods such as Wanda

Read original source ↗ Open in workspace

recordType

paper

region

Global