SOURCE-LINKED INTELLIGENCE

Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection

arXiv · AI, language, vision and robotics · article · Sep 13, 2026 · UTC

Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face co

Read original source ↗ Open in workspace

recordType

paper

region

Global