SOURCE-LINKED INTELLIGENCE

The Curse of Multilinguality in Lexical Normalization

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

Lexical normalization rewrites the noisy, non-standard words that fill user-generated text (tmrw, u, gr8) into their standard forms. Because labelled data is scarce for most languages, a popular shortcut is to train a single model on many languages at once. We ask a simple question: how many languages should such a model be trained on? Using one fixed-capacity character-level model and twelve languages from a standard benchmark, we vary the number of jointly trained languages from one to twelve and measure per-language accuracy. We find a clear curse of multilinguality: accuracy is highest whe

Read original source ↗ Open in workspace

recordType

paper

region

Global