arXiv:2509.04154 · v14
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
摘要
We introduce Robust Filter Attention (RFA), a formulation of self-attention as a robust state estimator. Each token is treated as a noisy observation of a latent trajectory governed by a linear stochastic differential equation (SDE), and attention weights are determined by consistency under this model rather than static feature similarity. Under isotropic noise and decay assumptions, RFA matches the computational complexity of standard attention. On language modeling benchmarks, RFA matches or improves on RoPE's perplexity within the training window while remaining stable under zero-shot extrapolation to longer contexts, where RoPE degrades sharply. The framework also provides a dynamical interpretation of standard positional mechanisms, connecting rotational embeddings and recency biases to transport and uncertainty propagation induced by dynamics.
作者备注：Corrects the RoPE baseline implementation (B1, B3, B4) and the M2.4 ablation; adds ablations M2.7 and M2.8. Main conclusions are unchanged: the in-window gain over RoPE is smaller and the extrapolation gain larger than previously reported