SOURCE-LINKED INTELLIGENCE

Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objective reduces the compute available for downstream adaptation. Despite its practical importance, this trade-off is not yet well understood theoretically, even in simple models. In this paper, we cast this allocation as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget, using regularised least squares trained by gradient descent as a tractable setting. We characterise the optimal split unde

Read original source ↗ Open in workspace

recordType

paper

region

Global