← 研踪 PaperTrailCVF · CVPR 2026:cvf:9a5025db796895edb69153e638f685aa · 记录 #1
Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
Vyas, ApoorvChang, Heng-JuiYang, Cheng-FuHuang, Po-YaoGao, LuyaRichter, JuliusChen, SanyuanLe, MatthewDollár, PiotrFeichtenhofer, ChristophLee, AnnHsu, Wei-Ning 来源日期(仅到年份) 2026
来源记录日期 2026
雷达观测 2026-11-05 04:29
摘要
We introduce Perception Encoder-Audiovisual, PE-AV, a new family of encoders for audio and video understanding trained with scaled contrastive learning. Building on PE, PE-AV makes several key contributions to extend representations to audio, and natively support joint embeddings across audio-video, audio-text, and video-text modalities. PE-AV's unified cross-modal embeddings enable novel tasks such as speech retrieval, and set a new state of the art across standard audio and video benchmarks. We unlock this by building a strong audiovisual data engine that synthesizes high-quality captions for O(100M) audio-video pairs, enabling large-scale supervision consistent across modalities. Our audio data includes speech, music, and general sound effects--avoiding single-domain limitations common in prior work. We exploit ten pairwise contrastive objectives, showing that scaling cross-modality and caption-type pairs strengthens alignment and improves zero-shot performance. Models and code are available.
论文发现出处
这些页面提供论文线索,不代表同行评审或质量认证。
观测到的版本
平台记录号代表本站观测到的元数据变化,不是作者修订号。来源日期仅到月份或年份时,按该月或该年首日存档。
记录 #1 · 2026 · 最新观测版本
Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
We introduce Perception Encoder-Audiovisual, PE-AV, a new family of encoders for audio and video understanding trained with scaled contrastive learning. Building on PE, PE-AV makes several key contributions to extend representations to audio, and natively support joint embeddings across audio-video, audio-text, and video-text modalities. PE-AV's unified cross-modal embeddings enable novel tasks such as speech retrieval, and set a new state of the art across standard audio and video benchmarks. We unlock this by building a strong audiovisual data engine that synthesizes high-quality captions for O(100M) audio-video pairs, enabling large-scale supervision consistent across modalities. Our audio data includes speech, music, and general sound effects--avoiding single-domain limitations common in prior work. We exploit ten pairwise contrastive objectives, showing that scaling cross-modality and caption-type pairs strengthens alignment and improves zero-shot performance. Models and code are available.
查看此版本解读 →查看此版本原文 ↗