SOURCE-LINKED INTELLIGENCE

Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Per-matrix singular value decomposition (SVD) truncation is Eckart-Young optimal in the whitened Frobenius norm, but errors from independently compressed matrices compound through the block's nonlinear forward pass. Inspired in part by hierarchical variational optimization in quantum many-body methods, we introduce a three-level chain that widens optimization scope from individual matrices to Transformer blocks to the full model: whitened SVD~(L1), block-level joint optimization~(L2), and end-to-end language-modeling loss refinement~(L3), all from 256 calibration sequences, with no instruction

Read original source ↗ Open in workspace

recordType

paper

region

Global