SOURCE-LINKED INTELLIGENCE

When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents

arXiv · AI, language, vision and robotics · article · Sep 9, 2026 · UTC

LLM agents increasingly rely on external skills retrieved at runtime, making skill selection from large repositories a critical challenge. We present a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data. We found that the synthetic-data fine-tuning improves in-distribution retrieval but it causes catastrophic forgetting on real and out-of-distribution (OOD) data. We evaluate several forgetting mitigation fine-tuning approaches inspired by continual learning, including embedding-anchor regularization, Learning

Read original source ↗ Open in workspace

recordType

paper

region

Global