SOURCE-LINKED INTELLIGENCE

Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone Distillation

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

A compressed student has two shapes that need not agree: the weight it deploys at inference and the weight family its training can reach. We show that a state-of-the-art weight-inheritance distiller, Low-Rank Clone (LRC), deploys a full-width student MLP but ties training to a teacher-induced slice, leaving 62.5-81.4% of each deployed matrix's independent linear degrees of freedom unreachable-paid for at inference, never trainable. Our principle is one line: train what you deploy. From the identical LRC warm start, we make the training object the entire deployed matrix, with no change in deplo

Read original source ↗ Open in workspace

recordType

paper

region

Global