跳到正文
研踪PaperTrail

论文库

检索计算机领域论文,按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+1858新增论文
+1473其中 AI 相关
+199已有论文修订
+59新增 AI 导读

arXiv 最近完成扫描:2026-11-18 04:28 · 全站最近入库:2026-11-18 11:33各信源进度 →

统计口径与数据新鲜度

统计 2026-11-17 11:40 — 2026-11-18 11:40(北京时间) · 点击数字查看对应简报;AI 相关包含在新增论文内。

最近完成检查:2026-11-18 04:28最近发现新增版本:2026-11-18 11:33库内论文最新更新时间:2026-11-17 20:55

每分钟刷新已保存的数据。空检查不代表采集异常;新收录可能包含历史论文。AI 导读按新增版本计数。

25,964 已收录论文21,631 AI 相关7,801 有修订112 HF 社区发现

浏览论文库

时间范围跟随排序:版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常,查看状态 →

找到 3 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-18 04:27(北京时间)
版本更新 · v2arXiv:2604.12151

Distinct mechanisms underlying in-context learning in transformers

Cole Gibson, Wenping Cui, Gautam Reddy

原文摘要

Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems. Here, we provide a complete mechanistic characterization of this behavior in transformers trained on a finite set $S$ of discrete Markov chains. The transformer displays four algorithmic phases, characterized by whether the network memorizes and generalizes, and whether it uses 1-point or 2-point statistics. We show that the four phases are implemented by multi-layer subcircuits that exemplify two qualitatively distinct mechanisms for implementing context-adaptive computations. Minimal models isolate the key features of both motifs. Memorization and generalization phases are delineated by two boundaries that depend on data diversity, $K = |S|$. The first ($K_1^\ast$) is set by a kinetic competition between subcircuits and the second ($K_2^\ast$) is set by a representational bottleneck. A symmetry-constrained theory of a transformer's training dynamics explains the sharp transition from 1-point to 2-point generalization and identifies key features of the loss landscape that allow the network to generalize. Put together, we show that transformers develop distinct subcircuits to implement in-context learning and identify conditions that favor certain mechanisms over others.

cs.LG · cond-mat.dis-nn · cond-mat.stat-mech

版本更新 · v2arXiv:2610.25596

Cognitive Maps in Large Language Models

Albert Qin, Jonathan D. Cohen, Gautam Reddy

原文摘要

It has long been proposed that animals construct cognitive maps to support flexible behavior in novel environments. We tested whether large language models form analogous internal representations using relational reasoning tasks and causal "mosaic" experiments. Models beyond the ten-billion-parameter scale inferred unstated relationships and generalized to synthetic entity names. Across models and tasks, models formed abstract representations of relational structure, additively combining them with representations of entity identity. Relationships were primarily stored as pairwise links and composed retrospectively, rather than prospectively encoded in a global coordinate system. A sparse, distributed set of key-value pairs mediated binding; perturbing them implanted "false memories" that produced predictable behavioral changes. Our findings reveal a conserved organization analogous to a cognitive map supporting flexible and systematic reasoning in language models.

cs.CL · cs.LG

新论文 · v1arXiv:2611.01564

Meta-memorization and memorization scaling laws in transformers

Alex Nguyen, Kenneth A. Norman, Gautam Reddy

原文摘要

Larger transformers can memorize more data, with a capacity (in bits) that scales roughly linearly with the number of parameters. Counterintuitively, larger models also memorize faster, requiring fewer repetitions of a sequence during training before it is memorized. We reproduce both memorization scaling laws in transformers trained on random binary sequences, showing that rapid memorization can arise even when data contain no semantic or compositional structure. Rapid memorization is due to a meta-memorization effect, where models trained to memorize a large set of random sequences learn to memorize novel sequences more quickly. The rate $r(t)$ at which models memorize grows with training time $t$ as a power-law $r(t) \sim t^β$ with a fixed exponent $β$ but an increasing upper cutoff across model scales. Meta-memorization enables even one-layer transformers to few-shot memorize a 100-bit sequence. This effect is partly explained by the formation of structured attention heads that convey position-invariant sequence information to MLPs where memorized sequences are stored. Meta-memorization is recapitulated in a simplified model of a dense associative memory where retrieval is constrained by an attentional bottleneck. Our results suggest transformers, through pre-training, acquire reusable mechanisms that enable highly efficient storage and retrieval of memories.

cs.LG · cs.AI