← 研踪 PaperTrailarXiv:2609.18078 · v2
Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability
首次提交 2026-09-16 11:28
版本更新 2026-10-27 10:46
雷达观测 2026-10-29 00:35
摘要
Warm-start transfer accelerates delayed generalization (grokking) on algorithmic tasks, yet which architectural components drive this acceleration and whether transferred circuits remain stable under continued optimization remain unclear. Across a scale-matched 108-run factorial battery ($N=12$ paired seeds), transferring internal attention/MLP matrices ($B$) alongside representations ($E+U$) lifts early accuracy by $5.46$ percentage points and cuts validation confirmation latency by $617$ steps (Holm $p=0.0039$). Whereas readout plus internal blocks ($U+B$) matches full transfer within $\pm 500$ steps in 1-layer models (TOST $p=2.91\times 10^{-13}$), a 2-layer replication confirms internal-block acceleration ($+704.67$ units, $p=4.88\times 10^{-4}$; $8.1\times$ faster trigger) while exposing an inter-block interface where omitting embeddings incurs a $-4{,}475.6$-unit deficit (TOST $p=1.00$). Under continued optimization, unshielded runs suffer severe post-grokking relapse. Freezing carriers ($E+U$) suppresses 1-layer drawdown ($22.06\% \to 0.60\%$ on $2a+b$; $28.56\% \to 5.09\%$ on $3a+b$; $11.09\% \to 0.15\%$ on $a^2+b^2$), but leaves internal blocks vulnerable at high step sizes. Dissecting 12 post-trigger schedules shows that Curvature-Preconditioned Bounded Step-Budget Consolidation ($S_H \approx 0.02, λ=1.0$, preserved $v_t$)---via 201-step micro-stepping ($86.6\%$ lower active compute) or exponential decay ($τ=200$)---compresses drawdown to $0.08\%$--$0.17\%$ on 1-layer $2a+b/3a+b$, $0.13\%$--$0.20\%$ on 2-layer $2a+b$, $0.02\%$ ($66.81\% \to 0.02\%$, Holm $p=0.0020$) on unseen pure-FP32 2-layer subtraction ($a-b$), and $0.09\%$--$0.18\%$ (Holm $p=0.031$) on non-abelian $S_5$ where carrier freezing fails ($10.73\%$ drawdown, $p=0.273$).
作者备注：19 pages, 6 figures, 9 tables, conference preprint
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-27 10:46 · 最新观测版本
Beyond Embedding Transfer: Component Roles in Grokking Transfer and Stability
Warm-start transfer accelerates delayed generalization (grokking) on algorithmic tasks, yet which architectural components drive this acceleration and whether transferred circuits remain stable under continued optimization remain unclear. Across a scale-matched 108-run factorial battery ($N=12$ paired seeds), transferring internal attention/MLP matrices ($B$) alongside representations ($E+U$) lifts early accuracy by $5.46$ percentage points and cuts validation confirmation latency by $617$ steps (Holm $p=0.0039$). Whereas readout plus internal blocks ($U+B$) matches full transfer within $\pm 500$ steps in 1-layer models (TOST $p=2.91\times 10^{-13}$), a 2-layer replication confirms internal-block acceleration ($+704.67$ units, $p=4.88\times 10^{-4}$; $8.1\times$ faster trigger) while exposing an inter-block interface where omitting embeddings incurs a $-4{,}475.6$-unit deficit (TOST $p=1.00$). Under continued optimization, unshielded runs suffer severe post-grokking relapse. Freezing carriers ($E+U$) suppresses 1-layer drawdown ($22.06\% \to 0.60\%$ on $2a+b$; $28.56\% \to 5.09\%$ on $3a+b$; $11.09\% \to 0.15\%$ on $a^2+b^2$), but leaves internal blocks vulnerable at high step sizes. Dissecting 12 post-trigger schedules shows that Curvature-Preconditioned Bounded Step-Budget Consolidation ($S_H \approx 0.02, λ=1.0$, preserved $v_t$)---via 201-step micro-stepping ($86.6\%$ lower active compute) or exponential decay ($τ=200$)---compresses drawdown to $0.08\%$--$0.17\%$ on 1-layer $2a+b/3a+b$, $0.13\%$--$0.20\%$ on 2-layer $2a+b$, $0.02\%$ ($66.81\% \to 0.02\%$, Holm $p=0.0020$) on unseen pure-FP32 2-layer subtraction ($a-b$), and $0.09\%$--$0.18\%$ (Holm $p=0.031$) on non-abelian $S_5$ where carrier freezing fails ($10.73\%$ drawdown, $p=0.273$).
查看此版本解读 →查看此版本原文 ↗