跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.29526 · v2

Geometric state space, memory representation and architectures

首次提交 2026-10-21 18:14
版本更新 2026-11-18 03:14
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

We introduce a geometric framework for deriving attention architectures. A sequence forms a memory m∈Mnm\in M_n, and an attention pairing ρn ⁣:Mn→Hom⁡(A,A)ρ_n\colon M_n\to\operatorname{Hom}(A,A) determines how that memory acts on each token state. Different representations of this action recover familiar architectures: softmax attention reads individual key--value pairs, linear attention accumulates their contributions in matrix statistics, and Intention fits a regression map from keys to values. The framework also suggests new architectures by changing the geometry of the query--key pairing, the function carried by each value, or the space representing the memory. ImageNet-1K experiments illustrate these constructions through geometric CAFormer variants, a ViT with query-dependent values, and a linear ViT with a wider memory representation.

作者备注:14 pages

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 03:14 · 最新观测版本

Geometric state space, memory representation and architectures

We introduce a geometric framework for deriving attention architectures. A sequence forms a memory m∈Mnm\in M_n, and an attention pairing ρn ⁣:Mn→Hom⁡(A,A)ρ_n\colon M_n\to\operatorname{Hom}(A,A) determines how that memory acts on each token state. Different representations of this action recover familiar architectures: softmax attention reads individual key--value pairs, linear attention accumulates their contributions in matrix statistics, and Intention fits a regression map from keys to values. The framework also suggests new architectures by changing the geometry of the query--key pairing, the function carried by each value, or the space representing the memory. ImageNet-1K experiments illustrate these constructions through geometric CAFormer variants, a ViT with query-dependent values, and a linear ViT with a wider memory representation.

查看此版本解读 →查看此版本原文 ↗