← 研踪 PaperTrailarXiv:2610.24026 · v2
Dynamical Kernel Attention: Estimating the Present from a Record of the Past
首次提交 2026-10-16 12:26
版本更新 2026-11-17 16:02
雷达观测 2026-11-18 19:42
摘要
Self-attention encodes where a past token lies, but not how its uncertainty grows with time: evidence accumulates without bound as the context grows, and nothing indicates which tokens are safe to discard. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state governed by a stochastic differential equation. Under this model, each head draws bounded evidence from any context. This lets each head abstain when relevant evidence is weak, and gives each head a memory window outside which discarding tokens changes its output by a provably bounded amount. On language modeling, DKA outperforms RoPE within the training length and ALiBi far beyond it, and matches sliding-window attention with a substantially smaller key-value cache.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-17 16:02 · 最新观测版本
Dynamical Kernel Attention: Estimating the Present from a Record of the Past
Self-attention encodes where a past token lies, but not how its uncertainty grows with time: evidence accumulates without bound as the context grows, and nothing indicates which tokens are safe to discard. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state governed by a stochastic differential equation. Under this model, each head draws bounded evidence from any context. This lets each head abstain when relevant evidence is weak, and gives each head a memory window outside which discarding tokens changes its output by a provably bounded amount. On language modeling, DKA outperforms RoPE within the training length and ALiBi far beyond it, and matches sliding-window attention with a substantially smaller key-value cache.
查看此版本解读 →查看此版本原文 ↗