跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.22449 · v1

How Many Training Labels Are Wrong? Auditing a Fitted Model through Gradient Allocation

首次提交 2026-11-17 17:16
版本更新 2026-11-17 17:16
雷达观测 2026-11-18 19:42
arXiv ↗PDF ↗

摘要

How many labels in a training set are wrong, and which ones? The usual answers read a held-out set or the model's outputs. A held-out reading depends on how the held-out set was drawn, and an output-based reading fails once the model has fitted its wrong labels, because the loss of a fitted wrong label is small and the prediction agrees with it. We read the answer from the per-example gradients of a single trained model. Each example, together with its label, asks the network to change in a particular way. We normalise the magnitudes of its gradient into a gradient allocation and compare it with the mean allocation of every class. An example is flagged when relabelling it as its nearest other class makes its allocation fit better than under its own label, and the share of flagged examples is the reading of the noise rate. The rule has no threshold and needs no clean data, no held-out set and no further training. After the model has fitted 99-100% of its wrong labels, the allocation still separates wrong from right labels with an AUC of 0.94-0.96 on the noisiest human labels of CIFAR-10N, where a feature-space margin falls to 0.79-0.84 and the loss, the gradient norm and the self-influence of an example to 0.73-0.76. On such a fitted model, readings of the noise rate from predictions or feature margins fall to 0-6% at true rates of 8-58% on CIFAR-10 with ViT-S, while the audit reads 54-96% of the true rate, and at least 93% of the examples it counts carry wrong labels. On a briefly trained model, before the wrong labels are fitted, the reading lies within 1.7 percentage points of the true noise rate on CIFAR-10 from 0% to 58% noise. Confident learning, which trains four further models, separates wrong labels about as well (AUC 0.925-0.998 against 0.900-0.995) and mostly reads the noise rate more closely, at a lower precision. An audit of 5,000 examples takes 14 to 27 seconds on one GPU.

作者备注:14 pages, 8 tables, 1 figure. Preliminary version

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-17 17:16 · 最新观测版本

How Many Training Labels Are Wrong? Auditing a Fitted Model through Gradient Allocation

How many labels in a training set are wrong, and which ones? The usual answers read a held-out set or the model's outputs. A held-out reading depends on how the held-out set was drawn, and an output-based reading fails once the model has fitted its wrong labels, because the loss of a fitted wrong label is small and the prediction agrees with it. We read the answer from the per-example gradients of a single trained model. Each example, together with its label, asks the network to change in a particular way. We normalise the magnitudes of its gradient into a gradient allocation and compare it with the mean allocation of every class. An example is flagged when relabelling it as its nearest other class makes its allocation fit better than under its own label, and the share of flagged examples is the reading of the noise rate. The rule has no threshold and needs no clean data, no held-out set and no further training. After the model has fitted 99-100% of its wrong labels, the allocation still separates wrong from right labels with an AUC of 0.94-0.96 on the noisiest human labels of CIFAR-10N, where a feature-space margin falls to 0.79-0.84 and the loss, the gradient norm and the self-influence of an example to 0.73-0.76. On such a fitted model, readings of the noise rate from predictions or feature margins fall to 0-6% at true rates of 8-58% on CIFAR-10 with ViT-S, while the audit reads 54-96% of the true rate, and at least 93% of the examples it counts carry wrong labels. On a briefly trained model, before the wrong labels are fitted, the reading lies within 1.7 percentage points of the true noise rate on CIFAR-10 from 0% to 58% noise. Confident learning, which trains four further models, separates wrong labels about as well (AUC 0.925-0.998 against 0.900-0.995) and mostly reads the noise rate more closely, at a lower precision. An audit of 5,000 examples takes 14 to 27 seconds on one GPU.

查看此版本解读 →查看此版本原文 ↗