Influcoder:将解码器梯度影响排名蒸馏到编码器以实现数据归因
Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution
Dimitri Kachler, Damien Sileo, Pascal Denis
AI 速读 基于摘要
作者提出 Influcoder,将解码器模型上的梯度影响排名蒸馏到一个编码器中,以更快速、低成本地实现大规模数据归因。
cs.CL
查看原文摘要
With the growth of LLMs' (Large Language Models) capabilities, there has been an increasing push to curate high quality datasets by filtering samples in the training data. In general, Data Attribution (DA) methods aim to estimate how individual samples in a training dataset can precondition a model to generate certain outputs. As an example, one might be interested in which samples in the data could be the source of toxic behavior after training the LLM. Many methods quantify this conditioning through the paradigm of influence functions. While methods of this family are effective in its function, they lack the necessary processing speed and storage compactness to be practically implemented on large datasets. We propose a method, Influcoder, as a quick and cost-effective approach to influence-based Data Attribution at scale.