← 研踪 PaperTrailarXiv:2312.05586 · v3
Generalized Influence Functions for Better Model Change Estimates
首次提交 2023-12-09 22:17
版本更新 2026-11-09 16:54
雷达观测 2026-11-10 15:22
摘要
Influence functions (IFs) approximate the effect of removing or reweighting training data on a trained model, but their linear approximation often lacks the accuracy required for reliable post hoc model editing. We identify two culprits and propose generalized influence functions (GIFs) and pseudo-LiSSA (p-LiSSA) to address each in turn. First, classical IFs update all parameters indiscriminately, leading to unnecessary updates to data-irrelevant parameters. GIFs instead restrict edits to a data-relevant subspace and leave the remaining parameters fixed. We theoretically show that GIFs achieve a tighter error bound than classical IFs when the parameter subset is properly selected, and that effective subspace selection requires avoiding low-curvature directions while retaining a sufficiently large component of the target gradient. Second, LiSSA iterations are destabilized by negative Hessian eigenvalues, which existing methods handle via damping, at the cost of a biased inverse-Hessian-vector product (iHVP) estimate. p-LiSSA eliminates this bias by solving the restricted inverse-Hessian problem without such damping. We further show that the presence of low-curvature directions slows p-LiSSA convergence, and that the data-relevant subspace selected by GIFs mitigates this. Across five image and text unlearning benchmarks, GIFs update only 10% of the parameters yet preserve accuracy up to 2.12 percentage points better than IF-based baselines, while improving unlearning performance by up to 2.73 percentage points. Moreover, p-LiSSA improves restricted iHVP estimation by reducing the residual by 9.6 over existing iHVP baselines, while using 19.8% less memory with comparable computation time. Code is available at https://github.com/hslyu/GIF.
作者备注：Accepted at NeurIPS 2026. 34 pages, 7 figures, and 5 tables
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-09 16:54 · 最新观测版本
Generalized Influence Functions for Better Model Change Estimates
Influence functions (IFs) approximate the effect of removing or reweighting training data on a trained model, but their linear approximation often lacks the accuracy required for reliable post hoc model editing. We identify two culprits and propose generalized influence functions (GIFs) and pseudo-LiSSA (p-LiSSA) to address each in turn. First, classical IFs update all parameters indiscriminately, leading to unnecessary updates to data-irrelevant parameters. GIFs instead restrict edits to a data-relevant subspace and leave the remaining parameters fixed. We theoretically show that GIFs achieve a tighter error bound than classical IFs when the parameter subset is properly selected, and that effective subspace selection requires avoiding low-curvature directions while retaining a sufficiently large component of the target gradient. Second, LiSSA iterations are destabilized by negative Hessian eigenvalues, which existing methods handle via damping, at the cost of a biased inverse-Hessian-vector product (iHVP) estimate. p-LiSSA eliminates this bias by solving the restricted inverse-Hessian problem without such damping. We further show that the presence of low-curvature directions slows p-LiSSA convergence, and that the data-relevant subspace selected by GIFs mitigates this. Across five image and text unlearning benchmarks, GIFs update only 10% of the parameters yet preserve accuracy up to 2.12 percentage points better than IF-based baselines, while improving unlearning performance by up to 2.73 percentage points. Moreover, p-LiSSA improves restricted iHVP estimation by reducing the residual by 9.6 over existing iHVP baselines, while using 19.8% less memory with comparable computation time. Code is available at https://github.com/hslyu/GIF.
查看此版本解读 →查看此版本原文 ↗