跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.24026 · v2

Dynamical Kernel Attention: Estimating the Present from a Record of the Past

首次提交 2026-10-16 12:26
版本更新 2026-11-17 16:02
雷达观测 2026-11-18 19:42
arXiv ↗PDF ↗

摘要

Self-attention encodes where a past token lies, but not how its uncertainty grows with time: evidence accumulates without bound as the context grows, and nothing indicates which tokens are safe to discard. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state governed by a stochastic differential equation. Under this model, each head draws bounded evidence from any context. This lets each head abstain when relevant evidence is weak, and gives each head a memory window outside which discarding tokens changes its output by a provably bounded amount. On language modeling, DKA outperforms RoPE within the training length and ALiBi far beyond it, and matches sliding-window attention with a substantially smaller key-value cache.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-17 16:02 · 最新观测版本

Dynamical Kernel Attention: Estimating the Present from a Record of the Past

Self-attention encodes where a past token lies, but not how its uncertainty grows with time: evidence accumulates without bound as the context grows, and nothing indicates which tokens are safe to discard. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state governed by a stochastic differential equation. Under this model, each head draws bounded evidence from any context. This lets each head abstain when relevant evidence is weak, and gives each head a memory window outside which discarding tokens changes its output by a provably bounded amount. On language modeling, DKA outperforms RoPE within the training length and ALiBi far beyond it, and matches sliding-window attention with a substantially smaller key-value cache.

查看此版本解读 →查看此版本原文 ↗