What Score Distillation Transfers to Compact Dense Retrievers
Kirill Dubovikov, Martin Takac, Salem Lahlou
原文摘要
Compact retrievers are often specialized by distilling a large embedding model's similarity scores on hard negatives mined by the teacher model. We argue that this common setup calls for a different loss than the scale-dependent ones usually used, such as MarginMSE and PairMSE. When the teacher discards candidates scored too close to the labeled positive, the negatives it keeps get nearly identical scores. These losses then mostly ask the student to reproduce this tiny score range rather than the teacher's ranking. A per-row Pearson correlation loss, invariant to the shift and scale of the scores in each row (a query and its 32 candidates), avoids this failure, and removing PairMSE's scale term alone recovers most of Pearson's lead. We propose row scale dispersion, computed from the untrained student before training, to tell when this matters. With this loss, on rows mined around a labeled document, the teacher's scores substitute for relevance labels rather than add to them. Training on the binary label alone works as well, with the same loss or, for one student, a contrastive loss: the teacher's value lies in choosing the negatives. Without labels, the teacher's scores on its own top-ranked documents recover most of the labeled gain, largely through its top document as a pseudo-label. Experiments with two compact students on three BEIR datasets support both conclusions, including, for one student, comparisons with MarginMSE and a tuned listwise KL. We suggest a simple practical rule: distill teacher-mined negatives with a scale-free loss and, on such rows, use the teacher's scores where labels are missing rather than on top of them.
cs.IR