Geometric Factual Recall in Transformers
Shauli Ravfogel, Gilad Yehudai, Joan Bruna, Alberto Bietti
原文摘要
How do transformer language models memorize factual associations? A common account views internal weight matrices as an associative memory. That account requires parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer memorizes random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension is sufficient. Subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute from the embedding. We extend these results to the multi-hop setting---chains of relational queries such as ``Who is the mother of the wife of $x$?''---and provide constructions with and without chain-of-thought that exhibit a provable capacity--depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with the predicted structure. Causal experiments support the proposed mechanism: when subject embeddings are appropriately re-initialized, the frozen MLP transfers zero-shot to entirely new bijections, showing it has learned a generic selection mechanism rather than functioning as a key-value memory.
cs.CL