SOURCE-LINKED INTELLIGENCE

Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

arXiv · AI, language, vision and robotics · article · Sep 5, 2026 · UTC

Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To

Read original source ↗ Open in workspace

recordType

paper

region

Global