arXiv:2602.19143 · v3
Incremental Learning of Sparse Attention Patterns in Transformers
摘要
Transformers integrate information from multiple past positions through sparse attention patterns that copy from specific locations. We study how these patterns form during training in a high-order Markov-chain setting, where the next token depends on several groups of past positions with differing statistical importance. Learning proceeds in discrete stages, each marked by the emergence of a sparse attention pattern for the next most important group. The dynamics are initially competitive, with all attention heads focusing on the dominant pattern, but later become cooperative, with different heads specializing in distinct patterns. For a tractable surrogate, the dynamics reduce to those of a tensor factorization problem, enabling us to prove stagewise convergence. Empirically, the number of stages completed depends on training set size, suggesting a form of implicit complexity control through early stopping. Overall, we give a theoretical account of how structured attention patterns and head specialization emerge without an explicit curriculum.
作者备注：Published at ICML 2026. OpenReview: https://openreview.net/forum?id=vSRh1qU5sH. 55 pages, 21 figures. v3: revised proofs in the appendix, with explicit assumptions in Theorems 4 and 5
期刊信息：Proceedings of the 43rd International Conference on Machine Learning (ICML 2026)