SOURCE-LINKED INTELLIGENCE

Structural priors for data-efficient language learning

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-language data to induce useful priors for natural language. This approach is a form of weight initialization for multilingual language modeling. We evaluate transfer via next-token-prediction loss, weight shifts in the model, and downstream linguistic benchmarks. Several symbolic data types - notably music, probabilistic grammars, and cellular automata - yield lower language-modeling loss than random initialization. Thes

Read original source ↗ Open in workspace

recordType

paper

region

Global