SOURCE-LINKED INTELLIGENCE

Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention Perspective

arXiv · AI, language, vision and robotics · article · Sep 3, 2026 · UTC

Pause-token methods improve LLM reasoning by inserting special tokens into sequences. Prior work explains these gains through computational expressivity. However, there is relatively little investigation into the training dynamics of pause tokens. We explore how pause tokens reshape the training dynamics of fine-tuning. Two controlled pilots expose distinct asymmetries. On a synthetic continual-learning task, masked pauses overwrite a previously-learned distribution roughly 4x less at matched final adaptation (H1, mode retention); on a synthetic math-reasoning probe, the boundary-adjacent toke

Read original source ↗ Open in workspace

recordType

paper

region

Global