← 研踪 PaperTrailarXiv:2611.04902 · v1
Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization
首次提交 2026-11-03 21:47
版本更新 2026-11-03 21:47
雷达观测 2026-11-04 23:39
摘要
Recurrent memory requires learning useful updates, but attention added to assist learning can also compromise length generalization. We study bounded delta attention: causal attention over uncorrected update proposals whose bounded correction enters the recurrent update. Its implementation in TraceRelay combines additive phase recurrence with attention computed before a parallel scan. On prefix-supervised modulo-5, trained at lengths 1--40, three findings characterize its behavior at length 80. First, upper delta attention attains $70.81\pm8.26$% accuracy across seven seeds, versus $69.21\pm22.42$% for the local-attention recurrent baseline (mean $\pm$ sample SD): a 63.2% reduction in SD with a similar mean. Upper state attention also reduces variation, reaching $74.75\pm6.24$%, so this benefit is not unique to update-level attention. Disabling the upper delta correction retains $70.62\pm8.60$%. Second, blocking gradients at the delta-attention inputs changes accuracy to $65.25\pm12.40$% in the matched seven-seed comparison, increasing SD by 50.2%. It also delays a 99% validation threshold on average among the same six seeds that reach it; the 95% threshold is reached earlier on average with input-gradient blocking across all seven seeds. Third, lower delta attention exhibits substantially less extrapolation degradation than lower state attention: 62.60% versus 19.40%, against a 71.06% baseline. Removing its correction reduces accuracy to 22.56%; its advantage over lower state attention also persists under input-gradient blocking. These observations motivate bounded update-level attention as a recurrent learning scaffold, combining reduced seed variation with less severe degradation in a placement where state attention fails.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-03 21:47 · 最新观测版本
Attending to Memory Writes Reduces Seed Variability in Recurrent Length Generalization
Recurrent memory requires learning useful updates, but attention added to assist learning can also compromise length generalization. We study bounded delta attention: causal attention over uncorrected update proposals whose bounded correction enters the recurrent update. Its implementation in TraceRelay combines additive phase recurrence with attention computed before a parallel scan. On prefix-supervised modulo-5, trained at lengths 1--40, three findings characterize its behavior at length 80. First, upper delta attention attains $70.81\pm8.26$% accuracy across seven seeds, versus $69.21\pm22.42$% for the local-attention recurrent baseline (mean $\pm$ sample SD): a 63.2% reduction in SD with a similar mean. Upper state attention also reduces variation, reaching $74.75\pm6.24$%, so this benefit is not unique to update-level attention. Disabling the upper delta correction retains $70.62\pm8.60$%. Second, blocking gradients at the delta-attention inputs changes accuracy to $65.25\pm12.40$% in the matched seven-seed comparison, increasing SD by 50.2%. It also delays a 99% validation threshold on average among the same six seeds that reach it; the 95% threshold is reached earlier on average with input-gradient blocking across all seven seeds. Third, lower delta attention exhibits substantially less extrapolation degradation than lower state attention: 62.60% versus 19.40%, against a 71.06% baseline. Removing its correction reduces accuracy to 22.56%; its advantage over lower state attention also persists under input-gradient blocking. These observations motivate bounded update-level attention as a recurrent learning scaffold, combining reduced seed variation with less severe degradation in a placement where state attention fails.
查看此版本解读 →查看此版本原文 ↗