← 研踪 PaperTrailarXiv:2602.05026 · v2
Laws of Learning Dynamics and the Core of Learners
首次提交 2026-02-05 04:27
版本更新 2026-11-10 00:57
雷达观测 2026-11-10 15:22
摘要
We formulate an algorithm-independent theory of learning dynamics for individual models and ensembles. We establish a conservation law, the vanishing of predictive entropy under successful learning, quantitative entropy and core-expansion bounds, and calibration and finite-sample validation results for identifying reliable predictions. These results motivate a two-generation ensemble that uses Jensen-Shannon (JS) disagreement to route uncertain inputs to specialized learners. CIFAR-10 experiments examine controlled perturbations, distribution shifts, and adversarial attacks. On a heterogeneous test environment, routing improves accuracy over probability averaging while keeping core accuracy near 95%. For adversarial defense, we consider a query-only black-box setting: the attacker can query the overall prediction function but does not know a particular internal decomposition into learners and routing rules. Our current attacks use a fixed-initial-route gradient, which is accurate for a single-step attack when the route remains unchanged, but may deviate from actual multi-step attacks, especially for large perturbations. We evaluate our two-layer defense against this approximation; whether additional adversarially trained layers improve resistance to stronger multi-step attacks remains for future study.
作者备注：39 pages
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-10 00:57 · 最新观测版本
Laws of Learning Dynamics and the Core of Learners
We formulate an algorithm-independent theory of learning dynamics for individual models and ensembles. We establish a conservation law, the vanishing of predictive entropy under successful learning, quantitative entropy and core-expansion bounds, and calibration and finite-sample validation results for identifying reliable predictions. These results motivate a two-generation ensemble that uses Jensen-Shannon (JS) disagreement to route uncertain inputs to specialized learners. CIFAR-10 experiments examine controlled perturbations, distribution shifts, and adversarial attacks. On a heterogeneous test environment, routing improves accuracy over probability averaging while keeping core accuracy near 95%. For adversarial defense, we consider a query-only black-box setting: the attacker can query the overall prediction function but does not know a particular internal decomposition into learners and routing rules. Our current attacks use a fixed-initial-route gradient, which is accurate for a single-step attack when the route remains unchanged, but may deviate from actual multi-step attacks, especially for large perturbations. We evaluate our two-layer defense against this approximation; whether additional adversarially trained layers improve resistance to stronger multi-step attacks remains for future study.
查看此版本解读 →查看此版本原文 ↗