← 研踪 PaperTrailarXiv:2602.08099 · v2
VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
首次提交 2026-02-09 03:39
版本更新 2026-11-03 21:55
雷达观测 2026-11-04 23:39
摘要
Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for text-vision tasks, typically by fine-tuning them to produce universal representations. However, their performance on video-text retrieval often remains below that of dedicated Video Foundation Models. In this paper, we revisit the video-text retrieval capabilities of MLLMs. We first analyze the zero-shot retrieval capabilities of pretrained MLLMs and show that they already encode substantial retrieval-relevant information: combining intermediate-layer embeddings with a calibrated MLLM head yields strong performance without any training. To further exploit these capabilities, we introduce a lightweight text-based optimization strategy that requires no paired multimodal data. Our method uses dense video captions as rich textual surrogates for video semantics and short summaries as compact, query-like views, showing that a carefully designed text-alignment objective can improve multimodal alignment through the shared MLLM backbone. We backup these findings through in-depth analysis. Remarkably, our lightweight text-only optimization outperforms its counterpart trained with paired video-text supervision, which is substantially more computationally expensive. Furthermore, our method outperforms existing approaches, often by a substantial margin, and achieves state-of-the-art performance across common video-text retrieval benchmarks. More broadly, our findings highlight the untapped retrieval capabilities of pretrained MLLMs and establish text-only optimization as a more efficient and effective substitute for large-scale visual supervision.
作者备注：NeurIPS 2026. Project page: https://iyttor.github.io/VidVec/
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-03 21:55 · 最新观测版本
VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for text-vision tasks, typically by fine-tuning them to produce universal representations. However, their performance on video-text retrieval often remains below that of dedicated Video Foundation Models. In this paper, we revisit the video-text retrieval capabilities of MLLMs. We first analyze the zero-shot retrieval capabilities of pretrained MLLMs and show that they already encode substantial retrieval-relevant information: combining intermediate-layer embeddings with a calibrated MLLM head yields strong performance without any training. To further exploit these capabilities, we introduce a lightweight text-based optimization strategy that requires no paired multimodal data. Our method uses dense video captions as rich textual surrogates for video semantics and short summaries as compact, query-like views, showing that a carefully designed text-alignment objective can improve multimodal alignment through the shared MLLM backbone. We backup these findings through in-depth analysis. Remarkably, our lightweight text-only optimization outperforms its counterpart trained with paired video-text supervision, which is substantially more computationally expensive. Furthermore, our method outperforms existing approaches, often by a substantial margin, and achieves state-of-the-art performance across common video-text retrieval benchmarks. More broadly, our findings highlight the untapped retrieval capabilities of pretrained MLLMs and establish text-only optimization as a more efficient and effective substitute for large-scale visual supervision.
查看此版本解读 →查看此版本原文 ↗