LACS: Distribution-Sensitive Label-Agnostic Confidence Selection for Dynamic Data Pruning
Heejin Choi
原文摘要
Training on large datasets motivates selecting which examples receive expensive training computation. Fixed quotas determine the amount of pruning in advance, while absolute confidence cutoffs depend on the score scale. We study a population-relative alternative that adapts the threshold and retained count together. We introduce LACS (Label-Agnostic Confidence Selector), a training-time rule based on maximum predictive confidence, requiring neither labels at selection time nor per-example gradients. LACS removes examples at or above a log-mean-exp reference plus a margin. We characterize its sensitivity to score gaps and mass, and derive an exact no-pruning condition. For $0<ρ<1$, the removed fraction is at most $ρ$ and at least one example remains. The unsigned score permits both confidently correct and confidently wrong examples to be removed. Seven evaluation settings include neural training, language modeling, and boosting. Neural execution studies reduce backward-active rows by 10-35%; on CIFAR-10, 29% fewer backward rows accompany accuracy means of 89.77% versus 89.82% for full CE. At a matched backward budget and number of optimizer updates, LACS is 1.0 pp above small-batch training and 1.2 pp above random selection over five seeds. Across 14 tabular datasets, quadratic boosting reduces mean test error from 23.36% to 22.96%. Its full training replays use about 30% fewer fitting rows; these counters span a different horizon from validation-selected accuracy. In one RBF-SVM case study, elapsed time falls by 37% with slightly lower error. These findings separate selective-work reduction from realized acceleration: the current exact-current-score neural implementation uses an additional scoring forward, so its backward sparsity does not yet translate into wall-clock acceleration. This overhead is an execution choice rather than a requirement of the selection rule.
cs.LG