← 研踪 PaperTrailarXiv:2605.12426 · v2
Geometric Factual Recall in Transformers
首次提交 2026-05-13 01:22
版本更新 2026-11-19 08:22
雷达观测 2026-11-20 10:50
摘要
How do transformer language models memorize factual associations? A common account views internal weight matrices as an associative memory. That account requires parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer memorizes random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension is sufficient. Subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute from the embedding. We extend these results to the multi-hop setting---chains of relational queries such as ``Who is the mother of the wife of ?''---and provide constructions with and without chain-of-thought that exhibit a provable capacity--depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with the predicted structure. Causal experiments support the proposed mechanism: when subject embeddings are appropriately re-initialized, the frozen MLP transfers zero-shot to entirely new bijections, showing it has learned a generic selection mechanism rather than functioning as a key-value memory.
作者备注:Accepted in Neurips 2026
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-19 08:22 · 最新观测版本
Geometric Factual Recall in Transformers
How do transformer language models memorize factual associations? A common account views internal weight matrices as an associative memory. That account requires parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer memorizes random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension is sufficient. Subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute from the embedding. We extend these results to the multi-hop setting---chains of relational queries such as ``Who is the mother of the wife of ?''---and provide constructions with and without chain-of-thought that exhibit a provable capacity--depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with the predicted structure. Causal experiments support the proposed mechanism: when subject embeddings are appropriately re-initialized, the frozen MLP transfers zero-shot to entirely new bijections, showing it has learned a generic selection mechanism rather than functioning as a key-value memory.
查看此版本解读 →查看此版本原文 ↗