SOURCE-LINKED INTELLIGENCE
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is being optimised vs. how it is being optimised. We disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokenise
Read original source ↗ Open in workspace
- recordType
- paper
- region
- Global
Evidence & attribution
- arXiv · AI, language, vision and robotics · 2026-09-16T17:59:45.000Z
- arXiv · Artificial Intelligence · 2026-09-16T17:59:45.000Z
First collected: 2026-09-19T20:26:32.566Z. This is not the publication date.