Gaspard:基于连续自回归流匹配与解耦无分类器引导的表达性统一语音与手势合成
Gaspard: Expressive unified speech and gesture synthesis with continuous autoregressive flow-matching and decoupled classifier-free guidance
Téo Guichoux, Théodor Lemerle, Catherine Pelachaud, Nicolas Obin
AI 速读 基于摘要
提出统一语音—手势生成模型 Gaspard,用连续自回归 next-step flow-matching 与解耦无分类器引导同时生成语音和共语手势,避免级联流程中语音独立于手势的问题。
cs.CV
查看原文摘要
Human communication involves highly coordinated multimodal signals such as speech and co-speech gestures. Existing approaches for generating both modalities commonly follow a cascaded pipeline, where speech is generated first and subsequently used to drive gesture generation; speech generation is therefore independent of gestures. We introduce Gaspard, a unified speech--gesture generation model based on continuous autoregressive generation with next-step flow-matching predictions, and trained on Seamless Interaction, a large-scale multimodal dataset. The heterogeneous temporal properties of speech and gesture motivate modality-specific prediction heads and decoupled classifier-free guidance over the two generation streams. This guidance rebalances the influence of speech and gesture at inference time, improving expressivity while mitigating dynamics collapse in continuous autoregressive gesture generation. Objective and perceptual evaluations show that Gaspard is a strong gesture generation model against unimodal baselines and that it produces highly synchronous speech and gestures compared to a matched cascaded system.