跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2603.29519 · v2

Beyond Dimensionality: Domain Shift and Catastrophic Forgetting in Single- vs. Multi-Vector Retrieval

首次提交 2026-03-31 18:04
版本更新 2026-11-19 23:00
雷达观测 2026-11-20 14:52
arXiv ↗PDF ↗

摘要

Recent work by Weller et al. introduced the naturalistic benchmark LIMIT and showed that widely used single-vector embedding models suffer substantial degradation in retrieval quality, attributing this primarily to limited embedding dimensionality. In our work, we revisit this explanation and argue that dimensionality alone is unlikely to account for the observed failures; in fact, results of Alon et al. imply that 2k+12k+1 dimensions suffice for top-kk retrieval over a fixed set of queries and documents. Motivated by this, we investigate alternative drivers of poor performance, including tokenization, linguistic structure, and domain shift. Our empirical analysis shows that tokenization artifacts and linguistic similarity yield only modest improvements, whereas domain shift and misalignment between embedding similarity and task-specific relevance are the dominant factors. While finetuning partially mitigates these issues and improves recall, it introduces catastrophic forgetting: finetuning single-vector models on LIMIT-like datasets reduces their MS MARCO Recall@10 by more than half, whereas multi-vector models forget much less. We observe the same pattern when finetuning on the real-world HotpotQA dataset, and between the single-vector and multi-vector heads of Jina-v4, which share one backbone.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-19 23:00 · 最新观测版本

Beyond Dimensionality: Domain Shift and Catastrophic Forgetting in Single- vs. Multi-Vector Retrieval

Recent work by Weller et al. introduced the naturalistic benchmark LIMIT and showed that widely used single-vector embedding models suffer substantial degradation in retrieval quality, attributing this primarily to limited embedding dimensionality. In our work, we revisit this explanation and argue that dimensionality alone is unlikely to account for the observed failures; in fact, results of Alon et al. imply that 2k+12k+1 dimensions suffice for top-kk retrieval over a fixed set of queries and documents. Motivated by this, we investigate alternative drivers of poor performance, including tokenization, linguistic structure, and domain shift. Our empirical analysis shows that tokenization artifacts and linguistic similarity yield only modest improvements, whereas domain shift and misalignment between embedding similarity and task-specific relevance are the dominant factors. While finetuning partially mitigates these issues and improves recall, it introduces catastrophic forgetting: finetuning single-vector models on LIMIT-like datasets reduces their MS MARCO Recall@10 by more than half, whereas multi-vector models forget much less. We observe the same pattern when finetuning on the real-world HotpotQA dataset, and between the single-vector and multi-vector heads of Jina-v4, which share one backbone.

查看此版本解读 →查看此版本原文 ↗