SOURCE-LINKED INTELLIGENCE

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

arXiv · AI, language, vision and robotics · article · Sep 11, 2026 · UTC

Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the trai

Read original source ↗ Open in workspace

recordType

paper

region

Global