SOURCE-LINKED INTELLIGENCE

Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

In the standard LM training pipeline, subword tokenisation is applied as a preprocessing step. Subword segmental language modelling is an alternative paradigm in which tokenisation is learned during training, allowing the model to discover subword units that optimise its training objective. In this paper, we present our submission to the 2026 BabyLM Challenge, for which we develop two new subword segmental LMs: SubSegGPT and SubSegDeBERTa. SubSegGPT is a decoder-only model that learns tokenisation during autoregressive pretraining. SubSegDeBERTa is an encoder-based model that jointly learns to

Read original source ↗ Open in workspace

recordType

paper

region

Global