arXiv:2611.19698 · v1
Data Attribution Should Steer Pretraining
摘要
Pretraining data curation decides what a model learns from, yet what makes a document worth learning from depends on what the model already knows. Influence-driven selectors already rank documents to improve one run; we argue that training-data attribution should steer pretraining by identifying testable data properties that become reusable curation criteria, evaluated on new data against cheaper selectors at matched total cost, discovery included. A document ranking is consumed by the run it was computed for, whereas an explained data property can be tested across corpora and checkpoints, especially where familiar quality signals diverge from measured usefulness. Because historical contribution, an understandable explanation, and future data value require distinct evidence, we specify a feedback loop that interprets attributed sources, tests candidate data changes, and states for five claim types what evidence each requires and what none establishes. We close with a falsifiable comparison against perplexity, classifier, and mixing baselines at equal total compute.
作者备注:Accepted to the 3rd Workshop on Attributing Model Behavior at Scale: Data Attribution and Provenance (ATTRIB 2026) at NeurIPS 2026, Idea Track. 4 pages main text, 13 pages total, 1 figure, 2 tables