跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2605.10395 · v2

Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

首次提交 2026-05-11 19:39
版本更新 2026-11-17 22:06
雷达观测 2026-11-18 19:41
arXiv ↗PDF ↗

摘要

We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width kk scales linearly with the input dimension dd -- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width} kck_c -- the number of learnable features at a given data budget nn -- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error εBO\varepsilon^{\rm BO} scales as n1/(2β)−1 n^{1/(2β)-1}, and a refinement regime in which it scales as n−1n^{-1}, where β>1/2β>1/2 is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation εBO=Θ(kcd/n)\varepsilon^{\rm BO}=Θ(k_c d/n). We further show empirically that a student trained with \textsc{Adam} near the effective width kck_c achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.

作者备注:Accepted for publication at NeurIPS 2026

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-17 22:06 · 最新观测版本

Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width kk scales linearly with the input dimension dd -- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width} kck_c -- the number of learnable features at a given data budget nn -- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error εBO\varepsilon^{\rm BO} scales as n1/(2β)−1 n^{1/(2β)-1}, and a refinement regime in which it scales as n−1n^{-1}, where β>1/2β>1/2 is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation εBO=Θ(kcd/n)\varepsilon^{\rm BO}=Θ(k_c d/n). We further show empirically that a student trained with \textsc{Adam} near the effective width kck_c achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.

查看此版本解读 →查看此版本原文 ↗