← 研踪 PaperTrailarXiv:2607.11465 · v3
What Score Distillation Transfers to Compact Dense Retrievers
首次提交 2026-07-13 20:18
版本更新 2026-11-11 19:10
雷达观测 2026-11-13 11:11
摘要
Compact retrievers are often specialized by distilling a large embedding model's similarity scores on hard negatives mined by the teacher model. We argue that this common setup calls for a different loss than the scale-dependent ones usually used, such as MarginMSE and PairMSE. When the teacher discards candidates scored too close to the labeled positive, the negatives it keeps get nearly identical scores. These losses then mostly ask the student to reproduce this tiny score range rather than the teacher's ranking. A per-row Pearson correlation loss, invariant to the shift and scale of the scores in each row (a query and its 32 candidates), avoids this failure, and removing PairMSE's scale term alone recovers most of Pearson's lead. We propose row scale dispersion, computed from the untrained student before training, to tell when this matters. With this loss, on rows mined around a labeled document, the teacher's scores substitute for relevance labels rather than add to them. Training on the binary label alone works as well, with the same loss or, for one student, a contrastive loss: the teacher's value lies in choosing the negatives. Without labels, the teacher's scores on its own top-ranked documents recover most of the labeled gain, largely through its top document as a pseudo-label. Experiments with two compact students on three BEIR datasets support both conclusions, including, for one student, comparisons with MarginMSE and a tuned listwise KL. We suggest a simple practical rule: distill teacher-mined negatives with a scale-free loss and, on such rows, use the teacher's scores where labels are missing rather than on top of them.
作者备注：updated preprint with new findings and experiments
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-11 19:10 · 最新观测版本
What Score Distillation Transfers to Compact Dense Retrievers
Compact retrievers are often specialized by distilling a large embedding model's similarity scores on hard negatives mined by the teacher model. We argue that this common setup calls for a different loss than the scale-dependent ones usually used, such as MarginMSE and PairMSE. When the teacher discards candidates scored too close to the labeled positive, the negatives it keeps get nearly identical scores. These losses then mostly ask the student to reproduce this tiny score range rather than the teacher's ranking. A per-row Pearson correlation loss, invariant to the shift and scale of the scores in each row (a query and its 32 candidates), avoids this failure, and removing PairMSE's scale term alone recovers most of Pearson's lead. We propose row scale dispersion, computed from the untrained student before training, to tell when this matters. With this loss, on rows mined around a labeled document, the teacher's scores substitute for relevance labels rather than add to them. Training on the binary label alone works as well, with the same loss or, for one student, a contrastive loss: the teacher's value lies in choosing the negatives. Without labels, the teacher's scores on its own top-ranked documents recover most of the labeled gain, largely through its top document as a pseudo-label. Experiments with two compact students on three BEIR datasets support both conclusions, including, for one student, comparisons with MarginMSE and a tuned listwise KL. We suggest a simple practical rule: distill teacher-mined negatives with a scale-free loss and, on such rows, use the teacher's scores where labels are missing rather than on top of them.
查看此版本解读 →查看此版本原文 ↗