SOURCE-LINKED INTELLIGENCE

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updat

Read original source ↗ Open in workspace

recordType

paper

region

Global