← 研踪 PaperTrailarXiv:2605.02206 · v3
Validity-Gated Evaluation of Multimodal Machine Unlearning
首次提交 2026-05-04 12:13
版本更新 2026-11-01 18:50
雷达观测 2026-11-04 23:43
摘要
Machine unlearning aims to remove the influence of training data without retraining a model from scratch. Evaluation of vision-language models relies on forget accuracy (FA), retain accuracy (RA), membership inference, and similarity to a retain-only reference, but evaluation pipelines are rarely tested. We propose a validity-gated unlearning audit (VGUA) with three checks: the forget set must be memorized without duplicate leakage into the retain set, the scorer must measure correctness, and the training code must implement the intended procedure. Applying these checks reveals that random MMBench splits leak duplicate questions, a lenient scorer hides memorization, and two coding errors remove the signal while training completes. After correcting these problems, we evaluate 234 unlearned models using six methods, three learning rates, and three backbones. Each model is compared with a retain-only reference trained using low-rank adaptation (LoRA). The results depend on update strength. Among models with RA at or above 0.70, lower FA generally indicates closer agreement with the retrained FA-RA operating point. Aggressive updates overshoot this point, collapse RA, and reverse method rankings. Relative to the original model, 78% of healthy models move closer to the retrained model's scores, whereas 94% move farther from it in activation space. Membership-inference advantage does not reliably track retraining alignment. Under option-shuffled training, unlearned models answer 25% of reordered forgotten ScienceQA questions, compared with 6% for the retain-only reference. These results show that multimodal unlearning conclusions depend on pipeline validity, update strength, and the comparison criterion.
作者备注：13 Pages , 3 figures, TNNLS
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-01 18:50 · 最新观测版本
Validity-Gated Evaluation of Multimodal Machine Unlearning
Machine unlearning aims to remove the influence of training data without retraining a model from scratch. Evaluation of vision-language models relies on forget accuracy (FA), retain accuracy (RA), membership inference, and similarity to a retain-only reference, but evaluation pipelines are rarely tested. We propose a validity-gated unlearning audit (VGUA) with three checks: the forget set must be memorized without duplicate leakage into the retain set, the scorer must measure correctness, and the training code must implement the intended procedure. Applying these checks reveals that random MMBench splits leak duplicate questions, a lenient scorer hides memorization, and two coding errors remove the signal while training completes. After correcting these problems, we evaluate 234 unlearned models using six methods, three learning rates, and three backbones. Each model is compared with a retain-only reference trained using low-rank adaptation (LoRA). The results depend on update strength. Among models with RA at or above 0.70, lower FA generally indicates closer agreement with the retrained FA-RA operating point. Aggressive updates overshoot this point, collapse RA, and reverse method rankings. Relative to the original model, 78% of healthy models move closer to the retrained model's scores, whereas 94% move farther from it in activation space. Membership-inference advantage does not reliably track retraining alignment. Under option-shuffled training, unlearned models answer 25% of reordered forgotten ScienceQA questions, compared with 6% for the retain-only reference. These results show that multimodal unlearning conclusions depend on pipeline validity, update strength, and the comparison criterion.
查看此版本解读 →查看此版本原文 ↗