SOURCE-LINKED INTELLIGENCE

Dynamic-Programming-Guided Hierarchical BPE and Empirical Analysis of Vocabulary Pruning

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

Byte Pair Encoding (BPE) constructs vocabularies through greedy pair merging, but the resulting merge order does not necessarily allocate a fixed model-visible vocabulary optimally for compression. We propose Dynamic-Programming-Guided Hierarchical BPE (DH-BPE), a vocabulary-construction method that combines token exposure under exact minimum-token segmentation with the hierarchical dependencies induced by BPE training. Starting from a modestly overshot BPE candidate vocabulary, DH-BPE uses dynamic programming to measure candidate utility and applies exposure-guided, dependency-aware pruning t

Read original source ↗ Open in workspace

recordType

paper

region

Global