← 研踪 PaperTrailarXiv:2611.19692 · v1
MathaMAGICal: Validating Trajectory Data Attribution During Capability Acquisition
首次提交 2026-11-15 08:51
版本更新 2026-11-15 08:51
雷达观测 2026-11-17 12:06
摘要
Influence-function-based methods like MAGIC provide exact first-order scores for trajectory data attribution, yet only retraining indicates whether those scores accurately predict data removal. We evaluate MAGIC on M- and M-parameter GPT-2-style models learning multistep arithmetic from scratch on filtered web text and four synthetic math families. Full-run attribution at M suffers from numerical overflow, and even with metagradient clipping, scores correlate near chance with retraining (linear datamodeling score, LDS, --). In contrast, short checkpoint windows yield stable, accurate predictions at both scales (LDS -- across four -step windows at M), whereas a single-checkpoint gradient-dot baseline shows near-zero correlation. Rankings of individual training blocks predict retraining only within that single run and do not transfer across seeds or hyperparameters (), whereas format-level patterns replicate across runs. Two data families, worked examples and primitive facts, occupy -- their corpus share in both the helpful and harmful score tails, reflecting strong bidirectional local sensitivity distinct from net benefit under multi-epoch family deletion. These results show that trajectory attribution is reliable within short, validated windows and yields family-level insights that generalize across runs.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-15 08:51 · 最新观测版本
MathaMAGICal: Validating Trajectory Data Attribution During Capability Acquisition
Influence-function-based methods like MAGIC provide exact first-order scores for trajectory data attribution, yet only retraining indicates whether those scores accurately predict data removal. We evaluate MAGIC on M- and M-parameter GPT-2-style models learning multistep arithmetic from scratch on filtered web text and four synthetic math families. Full-run attribution at M suffers from numerical overflow, and even with metagradient clipping, scores correlate near chance with retraining (linear datamodeling score, LDS, --). In contrast, short checkpoint windows yield stable, accurate predictions at both scales (LDS -- across four -step windows at M), whereas a single-checkpoint gradient-dot baseline shows near-zero correlation. Rankings of individual training blocks predict retraining only within that single run and do not transfer across seeds or hyperparameters (), whereas format-level patterns replicate across runs. Two data families, worked examples and primitive facts, occupy -- their corpus share in both the helpful and harmful score tails, reflecting strong bidirectional local sensitivity distinct from net benefit under multi-epoch family deletion. These results show that trajectory attribution is reliable within short, validated windows and yields family-level insights that generalize across runs.
查看此版本解读 →查看此版本原文 ↗