SOURCE-LINKED INTELLIGENCE

To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding and output matrices increase memory usage and slow inference, notably for small-scale models. It is also wasteful as models are often used for only a subset of languages. To address these issues, we introduce a modular framework for multilingual model training. First, we propose methods to learn large modular BPE and Unigram tokenizers that enable extraction of subtokenizers tailored to any langu

Read original source ↗ Open in workspace

recordType

paper

region

Global