SOURCE-LINKED INTELLIGENCE

TaRA: Training-Aware Low-Rank Adaptation Initialization

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA

Read original source ↗ Open in workspace

recordType

paper

region

Global