SOURCE-LINKED INTELLIGENCE

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Buildin

Read original source ↗ Open in workspace

recordType

paper

region

Global