SOURCE-LINKED INTELLIGENCE

Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Conversational speech depends on dialogue context and the listener's immediately preceding behavior. We propose ReACT-TTS, a two-stage framework that uses a one-second pre-response listener facial sequence to plan the next utterance's emotion and prosody before speech realization. On a strict dyadic MELD protocol, Temporal conditioning yields higher mean macro-F1 and VAD concordance than Text-only across ten seeds, while accuracy remains essentially unchanged. Ablations show that temporal modeling performs best among the visual variants and that an explicit early-to-late difference is unnecess

Read original source ↗ Open in workspace

recordType

paper

region

Global