SOURCE-LINKED INTELLIGENCE

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization. The step avoids expensive operations on full weight matrices, and its retraction is up to $2.8\times$

Read original source ↗ Open in workspace

recordType

paper

region

Global