跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2609.37745 · v2

Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

首次提交 2026-09-29 22:58
版本更新 2026-11-19 19:10
雷达观测 2026-11-20 19:57
arXiv ↗PDF ↗

摘要

Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a 1/31/3 exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the 1/31/3 account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size DD. We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to 1/31/3 across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents αrα_{r} and αtα_{t}. Under SGD, both are close to 1/31/3, so the data exponent is also 1/31/3 across different learning rates. Under Adam the two separate: αr≃0.48α_{r} \simeq 0.48 but αt≃0.08α_{t} \simeq 0.08. Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: DD-independent for SGD but falls with DD for Adam. Yet tuned to that optimum, the loss returns to D−1/3D^{-1/3} for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, 2αr+αt=12α_{r}+ α_{t} = 1, which fixes the optimal data exponent at 1/31/3. Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with D−1/3D^{-1/3} across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-19 19:10 · 最新观测版本

Optimizer-dependent training dynamics converge to the same one-third optimal data scaling

Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a 1/31/3 exponent emerges from learning peaked distributions. That account describes SGD, but models in practice are trained with adaptive optimizers. Here we separate two exponents the 1/31/3 account does not distinguish: how fast the loss falls with training steps along a single run, and how fast the optimally tuned loss falls with dataset size DD. We show that the first, a dynamic exponent, is optimizer-specific while the second, an optimal data exponent, converges to 1/31/3 across optimizers. In an online teacher-student model we decompose the loss into norm growth (radial) and alignment toward the teacher direction (tangential), each decaying as a power law with dynamic exponents αrα_{r} and αtα_{t}. Under SGD, both are close to 1/31/3, so the data exponent is also 1/31/3 across different learning rates. Under Adam the two separate: αr≃0.48α_{r} \simeq 0.48 but αt≃0.08α_{t} \simeq 0.08. Since the total loss is minimized when these two parts are balanced, the optimal learning rate is optimizer-dependent: DD-independent for SGD but falls with DD for Adam. Yet tuned to that optimum, the loss returns to D−1/3D^{-1/3} for both. A stochastic-dynamics analysis explains why: the optimizers can trade decay speed between the two channels, but they all fall on a single dynamic exponent relation, 2αr+αt=12α_{r}+ α_{t} = 1, which fixes the optimal data exponent at 1/31/3. Across seven optimizers, including Muon, the measured exponents are consistent with this relation, and the optimal-loss envelopes agree with D−1/3D^{-1/3} across them. The optimizer sets how fast a model learns per step; tuned optimally, it changes the prefactor but not the rate at which loss falls per sample.

查看此版本解读 →查看此版本原文 ↗