← 研踪 PaperTrailarXiv:2605.14659 · v2
Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning
首次提交 2026-05-14 18:12
版本更新 2026-11-03 20:26
雷达观测 2026-11-04 23:39
摘要
Once generalization becomes possible, does adding more data always make learning faster? In Needleman-Wunsch matrix generation, we find that small Transformers reach high exact-match validation accuracy in the fewest optimizer updates at an intermediate dataset size, while larger datasets converge more slowly. A random-suffix probe shows that, at larger dataset sizes, models fit the structured matrices before memorizing their random suffixes, suggesting that shared rule learning aids training-set fitting. To examine what makes rules easier to learn, we study elementary cellular automata (ECA), in which each binary cell updates from its own state and those of its two neighbors. These experiments identify lower-order predictive signals-output biases or predictive relationships involving only a few input bits-as a key factor in learnability. In Rule 90, finite-sample fluctuations create output biases and single-input dependencies absent from the uniform population. These signals weaken as the dataset grows. Biased training that introduces these signals accelerates generalization on an unchanged uniform validation distribution. These results support an explanation for faster convergence at intermediate dataset sizes: sampling fluctuations can facilitate rule acquisition, and larger datasets weaken this advantage.
作者备注：Accepted at NeurIPS 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-03 20:26 · 最新观测版本
Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning
Once generalization becomes possible, does adding more data always make learning faster? In Needleman-Wunsch matrix generation, we find that small Transformers reach high exact-match validation accuracy in the fewest optimizer updates at an intermediate dataset size, while larger datasets converge more slowly. A random-suffix probe shows that, at larger dataset sizes, models fit the structured matrices before memorizing their random suffixes, suggesting that shared rule learning aids training-set fitting. To examine what makes rules easier to learn, we study elementary cellular automata (ECA), in which each binary cell updates from its own state and those of its two neighbors. These experiments identify lower-order predictive signals-output biases or predictive relationships involving only a few input bits-as a key factor in learnability. In Rule 90, finite-sample fluctuations create output biases and single-input dependencies absent from the uniform population. These signals weaken as the dataset grows. Biased training that introduces these signals accelerates generalization on an unchanged uniform validation distribution. These results support an explanation for faster convergence at intermediate dataset sizes: sampling fluctuations can facilitate rule acquisition, and larger datasets weaken this advantage.
查看此版本解读 →查看此版本原文 ↗