SOURCE-LINKED INTELLIGENCE

Squeezing More from Limited Data with Recursive Transformers

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a larg

Read original source ↗ Open in workspace

recordType

paper

region

Global