← 研踪 PaperTrailarXiv:2611.19593 · v1
Learning Spatially Aware Audio-Visual Features from Videos with Mono Audio
首次提交 2026-11-15 03:57
版本更新 2026-11-15 03:57
雷达观测 2026-11-17 12:06
摘要
Comprehensive audio-visual perception requires both semantic and spatial reasoning. Existing spatial models suffer from poor generalization due to data scarcity, while semantic-only models fail to capture spatial characteristics completely. We introduce a model capable of jointly reasoning over binaural audio and vision on semantic and spatial tasks with improved generalization. To overcome the shortage in real-world data, we develop an automatic pipeline that localizes the sound source in a video, estimates its relative 3D trajectory, and renders binaural audio. We then use this to spatialize VGGSound, allowing us to train at a much larger scale. We demonstrate improved performance and generalization on spatial reasoning tasks while still remaining competitive on semantic tasks compared to semantic-only models.
作者备注:WACV 2027
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-15 03:57 · 最新观测版本
Learning Spatially Aware Audio-Visual Features from Videos with Mono Audio
Comprehensive audio-visual perception requires both semantic and spatial reasoning. Existing spatial models suffer from poor generalization due to data scarcity, while semantic-only models fail to capture spatial characteristics completely. We introduce a model capable of jointly reasoning over binaural audio and vision on semantic and spatial tasks with improved generalization. To overcome the shortage in real-world data, we develop an automatic pipeline that localizes the sound source in a video, estimates its relative 3D trajectory, and renders binaural audio. We then use this to spatialize VGGSound, allowing us to train at a much larger scale. We demonstrate improved performance and generalization on spatial reasoning tasks while still remaining competitive on semantic tasks compared to semantic-only models.
查看此版本解读 →查看此版本原文 ↗