← 研踪 PaperTrailarXiv:2603.29519 · v2
Beyond Dimensionality: Domain Shift and Catastrophic Forgetting in Single- vs. Multi-Vector Retrieval
首次提交 2026-03-31 18:04
版本更新 2026-11-19 23:00
雷达观测 2026-11-20 14:52
摘要
Recent work by Weller et al. introduced the naturalistic benchmark LIMIT and showed that widely used single-vector embedding models suffer substantial degradation in retrieval quality, attributing this primarily to limited embedding dimensionality. In our work, we revisit this explanation and argue that dimensionality alone is unlikely to account for the observed failures; in fact, results of Alon et al. imply that dimensions suffice for top- retrieval over a fixed set of queries and documents. Motivated by this, we investigate alternative drivers of poor performance, including tokenization, linguistic structure, and domain shift. Our empirical analysis shows that tokenization artifacts and linguistic similarity yield only modest improvements, whereas domain shift and misalignment between embedding similarity and task-specific relevance are the dominant factors. While finetuning partially mitigates these issues and improves recall, it introduces catastrophic forgetting: finetuning single-vector models on LIMIT-like datasets reduces their MS MARCO Recall@10 by more than half, whereas multi-vector models forget much less. We observe the same pattern when finetuning on the real-world HotpotQA dataset, and between the single-vector and multi-vector heads of Jina-v4, which share one backbone.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-19 23:00 · 最新观测版本
Beyond Dimensionality: Domain Shift and Catastrophic Forgetting in Single- vs. Multi-Vector Retrieval
Recent work by Weller et al. introduced the naturalistic benchmark LIMIT and showed that widely used single-vector embedding models suffer substantial degradation in retrieval quality, attributing this primarily to limited embedding dimensionality. In our work, we revisit this explanation and argue that dimensionality alone is unlikely to account for the observed failures; in fact, results of Alon et al. imply that dimensions suffice for top- retrieval over a fixed set of queries and documents. Motivated by this, we investigate alternative drivers of poor performance, including tokenization, linguistic structure, and domain shift. Our empirical analysis shows that tokenization artifacts and linguistic similarity yield only modest improvements, whereas domain shift and misalignment between embedding similarity and task-specific relevance are the dominant factors. While finetuning partially mitigates these issues and improves recall, it introduces catastrophic forgetting: finetuning single-vector models on LIMIT-like datasets reduces their MS MARCO Recall@10 by more than half, whereas multi-vector models forget much less. We observe the same pattern when finetuning on the real-world HotpotQA dataset, and between the single-vector and multi-vector heads of Jina-v4, which share one backbone.
查看此版本解读 →查看此版本原文 ↗