SOURCE-LINKED INTELLIGENCE

Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data

arXiv · AI, language, vision and robotics · article · Sep 16, 2026 · UTC

Large Language Models (LLMs) are now routinely trained using synthetic data, since high-quality human data has been exhausted by the ever increasing needs of larger and larger models. However, recursive training on synthetic data frequently induces model collapse, a degenerative feedback loop where models progressively forget the true underlying data distribution. Training on a mixture of synthetic and fresh human data is a logical countermeasure and can prevent model collapse. However, it is an open question as to what is the exact minimum required ratio of human-to-synthetic data to maintain

Read original source ↗ Open in workspace

recordType

paper

region

Global