SOURCE-LINKED INTELLIGENCE

Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling

arXiv · AI, language, vision and robotics · article · Sep 9, 2026 · UTC

When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliver comparable performance. We study Looped GPT-BERT in the BabyLM 2026 Strict-small setting, combining GPT-BERT's masked next-token and causal language-modeling objectives with depth-wise parameter sharing. We train on a preprocessed 7.48M-word English corpus and compare objective ratios, non-looped and looped architectures, and loop counts. Our final $4\times12$ model uses four physical layers for twelve recurrent t

Read original source ↗ Open in workspace

recordType

paper

region

Global