arXiv:2611.10691 · v1

A Geometric Theory for Nonequilibrium Transformer Dynamics

首次提交 2026-11-07 20:22

版本更新 2026-11-07 20:22

雷达观测 2026-11-10 11:14

摘要

We develop a variational and geometric theory based on statistical physics for the joint evolution of Transformer attention and token representations at fixed model parameters. To this end, we introduce a free energy for the attention and model the evolution of the attention weights by its Fisher--Rao gradient flow. At fixed token representations, we prove that the softmax matrix is the unique free-energy minimiser and that every positive attention trajectory converges to it. For prescribed time-dependent representations, we show that the attention dynamics becomes linear after a change to logit variables. This result yields tracking bounds relative to the instantaneous softmax and sufficient conditions for attention to converge to a fixed matrix. Hamiltonian dynamics for the representations and their momenta are then coupled to the attention relaxation driven by the same energy. This coupling results in an energy-dissipation law. We also develop two alternative Hamiltonians incorporating the attention value map. We prove global well-posedness of the attention flow and all three coupled systems.

作者备注：29 pages Submitted to Proc Roy Soc A