SOURCE-LINKED INTELLIGENCE

The Dynamics of Continuous Mixture Collapse in Language Models

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

LLMs latent-state reasoning methods replace discrete intermediate tokens with continuous states, such as weighted mixtures of token embeddings, to retain multiple possible reasoning directions rather than committing to one. Yet pretrained language models often fail to preserve these mixtures. We study why through a combination of theoretical analysis and controlled empirical investigations on a variety of models. We identify three independent, distinct sources of failure. First, transformer architectures already distort mixture geometry, and training substantially amplifies this effect. Moreov

Read original source ↗ Open in workspace

recordType

paper

region

Global