← 研踪 PaperTrailarXiv:2611.06970 · v1
DiaHeads: Jointly Generating Coordinated 3D Dialogue Heads with Non-verbal Cues from Dual Audio
首次提交 2026-11-04 20:36
版本更新 2026-11-04 20:36
雷达观测 2026-11-06 10:04
摘要
Audio-driven 3D head animation has made considerable progress in generation realism, yet generating coordinated behavior for two-person dialogue remains difficult, because current dialogue head generation methods overlook the modeling of non-verbal behavior semantics between the two participants. To address this limitation, we present DiaHeads, a framework that generates the behavior of both participants jointly. The two audio streams are concatenated and decoded together, so each participant's motion is conditioned on the other, which makes the interaction part of the generation itself. We further introduce four discrete non-verbal cues, gaze-on, head shaking, nodding, and smiling, and fuse them with the continuous parameters in a shared latent space through a product-of-experts posterior. These cues carry the interaction semantics and constrain how the continuous motion is generated. A flow-based prior over the shared latent variable replaces the standard Gaussian prior and makes the latent motion temporally dependent. These designs therefore yield coordinated 3D dialogue heads. To support this task we build CANDOR-FG, a dialogue corpus derived from CANDOR and augmented with synchronized FLAME parameters and 3D gaze for both interlocutors. Experiments on CANDOR-FG show that DiaHeads achieves competitive per-participant fidelity and substantial gains in interpersonal coordination over single-speaker and dyadic baselines. In addition, ablations confirm that joint generation, the flow-based prior, and multi-cue supervision are the sources of this coordination. Finally, a user study shows that the advantage of DiaHeads comes from perceived interactional coordination rather than from raw motion fidelity alone.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-04 20:36 · 最新观测版本
DiaHeads: Jointly Generating Coordinated 3D Dialogue Heads with Non-verbal Cues from Dual Audio
Audio-driven 3D head animation has made considerable progress in generation realism, yet generating coordinated behavior for two-person dialogue remains difficult, because current dialogue head generation methods overlook the modeling of non-verbal behavior semantics between the two participants. To address this limitation, we present DiaHeads, a framework that generates the behavior of both participants jointly. The two audio streams are concatenated and decoded together, so each participant's motion is conditioned on the other, which makes the interaction part of the generation itself. We further introduce four discrete non-verbal cues, gaze-on, head shaking, nodding, and smiling, and fuse them with the continuous parameters in a shared latent space through a product-of-experts posterior. These cues carry the interaction semantics and constrain how the continuous motion is generated. A flow-based prior over the shared latent variable replaces the standard Gaussian prior and makes the latent motion temporally dependent. These designs therefore yield coordinated 3D dialogue heads. To support this task we build CANDOR-FG, a dialogue corpus derived from CANDOR and augmented with synchronized FLAME parameters and 3D gaze for both interlocutors. Experiments on CANDOR-FG show that DiaHeads achieves competitive per-participant fidelity and substantial gains in interpersonal coordination over single-speaker and dyadic baselines. In addition, ablations confirm that joint generation, the flow-based prior, and multi-cue supervision are the sources of this coordination. Finally, a user study shows that the advantage of DiaHeads comes from perceived interactional coordination rather than from raw motion fidelity alone.
查看此版本解读 →查看此版本原文 ↗