跳到正文
研踪PaperTrail

论文库

检索计算机领域论文,按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+0新增论文
+0其中 AI 相关
+0已有论文修订
+92新增 AI 导读

arXiv 最近完成扫描:2026-11-23 01:07 · 全站最近入库:2026-11-21 12:19各信源进度 →

统计口径与数据新鲜度

统计 2026-11-22 03:30 — 2026-11-23 03:30(北京时间) · 点击数字查看对应简报;AI 相关包含在新增论文内。

最近完成检查:2026-11-23 01:07最近发现新增版本:2026-11-21 12:19库内论文最新更新时间:2026-11-20 02:59

每分钟刷新已保存的数据。空检查不代表采集异常;新收录可能包含历史论文。AI 导读按新增版本计数。

29,158 已收录论文24,181 AI 相关8,992 有修订112 HF 社区发现

浏览论文库

时间范围跟随排序:版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常,查看状态 →

找到 56 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-23 01:05(北京时间)
版本更新 · v3arXiv:2610.09186

模式识别与逐步推理的二分法

The Dichotomy Between Pattern Recognition and Step-by-Step Reasoning

Amrut Nadgir, Vijay Balasubramanian, Pratik Chaudhari

AI 速读 基于摘要

论文提出模式识别与逐步推理是同一谱系的两端,其区分取决于下一个词元所依赖的上下文长度,并证明逐步推理具有样本效率,且状态频率可平衡准确性与鲁棒性。

cs.LG · cond-mat.dis-nn

查看原文摘要

We argue that pattern recognition and step-by-step reasoning are two ends of a spectrum. A large language model (LLM) learns to reason step-by-step when data is structured such that the next token depends on a small amount of preceding context. Inference in LLMs resembles pattern recognition when the next token depends on a large amount of preceding context. If the next token depends on only the $c$ most recent tokens, reasoning traces are paths on a de Bruijn graph whose nodes are $c$-length contexts and edges are next-token transitions between contexts. The set of reasoning traces of a task forms a directed acyclic subgraph of the de Bruijn graph. An LLM that has learned all edges of this subgraph can compose them to solve longer, unseen tasks, i.e., it reasons step-by-step. We prove that the number of edges is vanishingly small compared to the number of reasoning traces. We also prove and show empirically that the number of training samples a transformer needs is a linear in the number of edges, so learning to reason step-by-step is sample efficient. We can induce de Bruijn structure in any task by maintaining a ``state'' that makes future reasoning independent of the past. The frequency of states in the reasoning trace determines $c$. We show, by fine-tuning Qwen2.5-1.5B-Instruct to solve equations and answer questions about stories, that frequent states (small $c$) result in higher accuracy but greater fragility to perturbations at test time. LLMs trained with a large $c$ are only as good as models that perform pattern recognition without reasoning. A moderate density of states balances accuracy and robustness. Real-world data has de Bruijn structure: Qwen3-14B and Qwen3-32B retain over 75\% of their accuracy on GSM8K, MATH-500 and GPQA-Diamond when attention is restricted to a sliding window less than 15\% as long as the full reasoning trace.

版本更新 · v3arXiv:2511.11927

低秩矩阵推断中的稀疏噪声:PCA基准

Sparse corruption in low-rank matrix inference: the PCA benchmark

Urte Adomaityte, Gabriele Sicuro, Pierpaolo Vivo

AI 速读 基于摘要

作者利用复制方法研究稀疏噪声下PCA的谱相变,发现两个信号强度阈值,并给出随机正则度分布下的精确闭式解。

stat.ML · cond-mat.dis-nn · cond-mat.stat-mech · cs.LG

查看原文摘要

Principal Component Analysis (PCA) is a standard tool for extracting a low-rank signal from noisy observations. It is known that when applying PCA to a rank-one signal corrupted by a dense, homogeneous noise, in the large matrix size limit, the BBP transition occurs, where the emergence of an outlying eigenvalue and the alignment of the corresponding eigenvector occur at the same critical signal strength. Here we study the case of sparse noise corruption. The noise matrix is modelled as the adjacency matrix of a weighted undirected graph with finite average connectivity. Using the replica method, we analytically compute the typical top eigenvalue, the top eigenvector component density, and the overlap with the signal, through recursive distributional equations solved by population dynamics. We identify two signal-strength transitions as functions of graph connectivity: $θ_{\rm crit}$, marking signal recovery by the top eigenvector and generalising the BBP transition, and $θ_{\rm b}$, where the signal-related eigenvalue detaches from the bulk. For noise with nonzero mean, these transitions need not coincide because of a structural sparse-graph outlier, leading to a discontinuous transition in the overlap with the top eigenvector. For a Random Regular degree distribution, we obtain exact closed-form expressions for both thresholds, the signal eigenvalue and the squared overlap, and show that the same formulas also predict the overlap of the signal with the second eigenvector in the intermediate regime $θ_{\rm b}<θ\leq θ_{\rm crit}$. For a Poissonian degree distribution, an approximate closure of the recursions yields fast and accurate predictions in the recovery phase that agree with exact results from population dynamics. We also recover dense-noise results in the large-connectivity limit, and validate the theory by numerical diagonalisation of large matrices.

新论文 · v1arXiv:2611.26228

注意力头专门化:样本复杂度中的相变

Attention Head Specialization: Phase Transitions in Sample Complexity

Léo Catteau, O. Duranthon, Lenka Zdeborová

AI 速读 基于摘要

作者通过实验与可解高维模型发现,注意力头专门化随训练数据量增加呈现相变,并区分统计与动力学起源。

cs.LG · cond-mat.dis-nn · stat.ML

查看原文摘要

Attention heads specialize into distinct functions during training, often emerging in stages together with sharp changes in model behavior. While this phenomenon has mostly been studied as a function of training time, we ask how head specialization depends on the amount of available training data. Empirically, we find that heads specialized to syntactic relations in BERT emerge as the dataset size increases. We then study this phenomenon in a solvable high-dimensional model of multi-head attention and establish that, in this model, head specialization is a genuine phase transition in sample complexity, corresponding to a non-analyticity in the test error and other observables. Depending on data structure and regularization, specialization can follow uninformative head differentiation or occur through a cascade of phase transitions that progressively recover weaker latent features, providing a statistical form of simplicity bias. Finally, we relate these statistical transitions to the stagewise emergence of specialized heads during online learning, disentangling statistical from dynamical origins of head specialization.

新论文 · v1arXiv:2611.25352

Compiling Constraints into Neuromorphic Ising Machines

Faiek Ahsan, Shantanu Chakrabartty

原文摘要

Most practical combinatorial optimization problems involve constraints, and an Ising machine that flips one spin at a time can respect them only through energy penalties that raise barriers between feasible solutions. Here we show that a higher-order neuromorphic Ising machine can enforce constraints through programmable transitions rather than penalties. In an autoencoder-like architecture, each inner-layer neuron represents a move, a family of multi-spin transitions that maps every feasible state to another feasible state, receives the energy change of its move through connections that outer-layer neurons multiplicatively switch on or off, and updates the state through the same connections. A compiler turns constraints into this wiring with a number of auxiliary clauses that grows only polynomially for parity, cardinality, one-hot, partition and permutation constraints. Restricting the search to feasible states reduces constraint handling to move design, and we show that the fastest-annealing moves are not those that mix fastest but those that lower the deepest energy barriers, and that moves chosen from the objective lower them further than random ones. On third-order graph matching, the machine needs orders of magnitude fewer iterations than penalty-based neuromorphic machines. Configured as a sampler on a field-programmable gate array, it decodes the surface code with maximum-likelihood accuracy up to code distance 15 within 52 $μ$s per shot and, under noisy syndrome measurements, decodes in real time for millisecond measurement rounds up to distance 9 with fewer logical errors than belief-propagation decoders.

cs.ET · cond-mat.dis-nn · cs.AR · cs.NE · quant-ph

新论文 · v1arXiv:2611.25124

Learning Integer Winding Evolution Times in Quantum Phase Estimation

Elijah Pelofske, Stephan Eidenbenz

原文摘要

Learning a Hamiltonian evolution time in Quantum Phase Estimation that produces phase distributions with low Shannon entropy circumvents the fundamental limit of digitization error of phase, independent of the number of phase qubits of accuracy. By introducing an optimization loop to minimize the Shannon entropy of measured eigenvalues produced by a QPE quantum circuit, minimum eigenvalue computation accuracy is greatly improved, which we show numerically on quantum magnet models. This technique works because standard QPE has a natural correspondence between entropy of the phase distribution and the analog accuracy of the minimum eigenvalue phase. This introduces for the first time a type of parameter learning to QPE, which makes near term practicality of QPE for fundamental sciences more feasible by using shorter depth QPE circuits.

quant-ph · cond-mat.dis-nn · cs.IT

新论文 · v1arXiv:2611.24089

Quantum Annealing with a Quantum Neural Network: Width of Annealing Matters

Mojtaba Asadollahi, Danial Khajehzadeh, Abolfazl Ramezanpour

原文摘要

Quantum annealing is a promising quantum algorithm for dealing with computationally hard optimization problems. Currently available hardware can implement the algorithm for a significant class of Hamiltonian problems. However, its performance is especially limited in discontinuous quantum phase transitions, where the energy gap shrinks exponentially with system size. In this paper, we introduce an extension of quantum annealing to a quantum neural network and study its performance in a number of prototypical optimization problems. In particular, we observe that already a random quantum annealing network of width W = 2 can exponentially reduce the number of evolution layers or depth of the annealing process in the quantum search problem. By tuning the network parameters the search problem can be solved with exponentially small errors by a quantum perceptron of depth L = 1 and width W = 2. The findings show that width of a quantum annealing network is critical to enhance the success probability of the algorithm by providing a larger and learnable evolution space for the annealing process.

quant-ph · cond-mat.dis-nn · cs.LG

版本更新 · v2arXiv:2611.11826

How Path Integral Diffusion Decides a Finite Spin Glass

Michael Chertkov

原文摘要

When an AI diffusion-type generator samples a frustrated spin glass, which macroscopic alternatives does it decide first, and when? We study this for bridge-diffusion models of the Path Integral Diffusion (PID) type started from a point source. We show that every isotropic PID is a time-changed Gaussian stochastic localization, and conversely every signal-to-noise schedule is realized by a PID protocol. What a continuous-time generator decides therefore does not depend on the protocol, which only sets when and at what cost. We describe the generator's state by its partial commitments: the probabilities it assigns, at each moment, to nested classes of configurations fixed by the target. The partial commitments are martingales that move only along class contrasts. Their ensemble average gives the information about each level at an exact rate set by squared magnetization contrasts. In the balanced, effectively binary approximation, a split reaches half of its class information near γ=1.044/λ_P, where λ_P is the leading eigenvalue of the parent covariance. The theory is applied to Bethe-lattice and Sherrington--Kirkpatrick (SK) glasses with up to 24 spins. It is observed generically that a trajectory first shifts its weight to one of the two mirror halves of the zero-field measure and then to the glassy classes inside it. These classes appear near the glass transition, keep their identity on cooling and are absent in a ferromagnet on the same graphs. Sparse (Bethe) and dense (SK) glasses differ in how often and how strongly such classes exist, not in how trajectories commit to them. For a generator that takes finitely many steps, the protocol matters through the late, local stage rather than the macroscopic decisions. Sampled posteriors at N=32,64 and small trained denoisers show the same sequence.

cond-mat.dis-nn · cond-mat.stat-mech · cs.AI · stat.ML

版本更新 · v3arXiv:2604.12151

Distinct mechanisms underlying in-context learning in transformers

Cole Gibson, Wenping Cui, Gautam Reddy

原文摘要

Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems. Here, we provide a mechanistic characterization of this behavior in transformers trained on a finite set $S$ of discrete Markov chains. The transformer displays four algorithmic phases, characterized by whether the network memorizes or generalizes, and whether it uses 1-point or 2-point statistics. We show that the four phases are implemented by multi-layer subcircuits that exemplify two qualitatively distinct mechanisms for implementing context-adaptive computations. Minimal models isolate the key features of both motifs. Memorization and generalization phases are delineated by two boundaries that depend on data diversity, $K = |S|$. The first ($K_1^\ast$) is set by a kinetic competition between subcircuits and the second ($K_2^\ast$) is set by a representational bottleneck. A symmetry-constrained theory of a transformer's training dynamics explains the sharp transition from 1-point to 2-point generalization and identifies key features of the loss landscape that allow the network to generalize. Put together, we show that transformers develop distinct subcircuits to implement in-context learning and identify conditions that favor certain mechanisms over others.

cs.LG · cond-mat.dis-nn · cond-mat.stat-mech

新论文 · v1arXiv:2611.23093

What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

Aritra Das, Francesco Cagnetta, Darshil Doshi, Victor V. Albert, Maissam Barkeshli, Andrey Gromov

原文摘要

Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $μ$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.

cs.CL · cond-mat.dis-nn · cs.AI · cs.LG

版本更新 · v2arXiv:2609.07456

Graph neural networks and the energetic cavity method for combinatorial optimization

Joe Bacchus George, George T. Cantwell

原文摘要

We study the use of graph neural networks (GNNs) for finding approximate ground states of Ising models. Efficiently finding these ground states is of broad significance because many combinatorial optimization problems can be formulated as an Ising model with the appropriate choice of couplings and fields. Exactly solving these problems is hard but there are many good heuristic methods. A lineage of these heuristics build from mean-field approximations: one approach uses the leading eigenvector of an appropriately defined matrix, another is the min-sum algorithm, also known as the energetic cavity method. Without modification, GNNs perform worse than both of these methods. We consider small modifications to the GNN to incorporate these heuristics and find that this considerably improves performance. While the modified approach is much stronger than other deep learning approaches, we still find that simulated annealing is generally as good for the same computational cost.

cond-mat.dis-nn · cs.LG

版本更新 · v2arXiv:2605.10395

Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

Minh-Toan Nguyen, Jean Barbier

原文摘要

We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width $k$ scales linearly with the input dimension $d$ -- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width} $k_c$ -- the number of learnable features at a given data budget $n$ -- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error $\varepsilon^{\rm BO}$ scales as $ n^{1/(2β)-1}$, and a refinement regime in which it scales as $n^{-1}$, where $β>1/2$ is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation $\varepsilon^{\rm BO}=Θ(k_c d/n)$. We further show empirically that a student trained with \textsc{Adam} near the effective width $k_c$ achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.

stat.ML · cond-mat.dis-nn · cond-mat.stat-mech · cs.IT · cs.LG

版本更新 · v5arXiv:2511.23152

Functional Compression as Occam's Razor: Algebraic Complexity from Sensitivity Minimization

Dongsung Huh, Lior Horesh, Halyun Jeong

原文摘要

Discovering discrete algebraic structures from data is a fundamental challenge in machine learning. We ask how continuous learning objectives can favor functional compression---representing data through shared algebraic composition rather than low-rank representations. We analyze the global optima of a canonical factorization model for algebraic completion (recovering finite binary operations) that demonstrated a preference for group operations. We measure algebraic complexity by the infimum of parameter sensitivity under exact reconstruction. We establish a universal complexity floor, whose attainment forces geometric alignment and unitarity of the latent representations. At the floor, input representations compose into the corresponding output representations; their reuse across the operation table propagates these local identities to enforce a globally shared algebraic structure. If the floor is not attained, a positive gap separates it from the feasible infimum. For cancellative operations, floor attainment forces the model's latent representation to be an injective homomorphism, equivalent to the target being isotopic to a group. In this case, every global minimizer is equivalent to the underlying group's unitary left-regular representation, up to isotopy and unitary gauge, resolving the central conjecture of Huh (2025). For non-cancellative operations, floor attainment forces exact structural redundancies (identical rows or columns) and is equivalent to a unitary factorization of the target's row- and column-reduced table with orthonormal output matrices, yielding the fewest distinct input matrices compatible with exact reconstruction. The resulting cancellative table then embeds in a group-isotope table. This analysis reveals a structural bias toward functional compression: at the sensitivity floor, algebraic composition of latent representations implements the target operation.

cs.LG · cond-mat.dis-nn · math.OC · math.RT · stat.ML

版本更新 · v4arXiv:2303.16880

Storage and Learning phase transitions in the Random-Features Hopfield Model

Matteo Negri, Clarissa Lauditi, Gabriele Perugini, Carlo Lucibello, Enrico M. Malatesta

原文摘要

The Hopfield model is a paradigmatic model of neural networks that has been analyzed for many decades in the statistical physics, neuroscience, and machine learning communities. Inspired by the manifold hypothesis in machine learning, we propose and investigate a generalization of the standard setting that we name Random-Features Hopfield Model. Here $P$ binary patterns of length $N$ are generated by applying to Gaussian vectors sampled in a latent space of dimension $D$ a random projection followed by a non-linearity. Using the replica method from statistical physics, we derive the phase diagram of the model in the limit $P,N,D\to\infty$ with fixed ratios $α=P/N$ and $α_D=D/N$. Besides the usual retrieval phase, where the patterns can be dynamically recovered from some initial corruption, we uncover a new phase where the features characterizing the projection can be recovered instead. We call this phenomena the learning phase transition, as the features are not explicitly given to the model but rather are inferred from the patterns in an unsupervised fashion.

cond-mat.dis-nn · cs.LG

版本更新 · v2arXiv:2610.41398

Distilling Knowledge from Search: Decoder Model Factory for Quantum LDPC Codes

Anil Bilgin, Wanda Hou, Pranav Srinivas Murali, Yi-Zhuang You

原文摘要

We introduce GraphMP, a learning-based message-passing decoder for quantum low-density parity-check codes that transfers without fine-tuning across code families, code sizes, physical error rates, and noise models. Its local message passing keeps computation scalable as the decoding graph grows. We further embed GraphMP in a beam-search algorithm that progressively refines candidate error patterns, trading additional decoding time for lower logical error rates. To turn this decoding effort into lasting improvements, we propose the decoder model factory, a paradigm for training successive models on data collected by earlier models during decoding. In our implementation, verified search trajectories provide reusable training data for distilling knowledge into subsequent models. Across generations, the model factory yields a search-enhanced decoder that is broadly competitive with Relay-BP. On the largest code block, which encodes $4612$ logical qubits, it achieves a lower logical error rate and about $20$ times the decoding throughput of the standard Relay-BP baseline. These results motivate a coupled quantum-classical architecture with two interacting cycles: rapid decoding protects quantum information, while slower learning adapts the models to changing noise conditions. The long-term vision is a scalable decoder model factory that continually produces high-performance, low-latency decoders for evolving quantum hardware.

quant-ph · cond-mat.dis-nn · cs.AI

版本更新 · v2arXiv:2611.00921

Solvable dynamical model of in-context learning

Ruoran Bai, Haiping Huang

原文摘要

Pretrained Transformer architectures display a surprising in-context learning ability without additional training. How the learning process guides the network to achieve this intriguing capability remains unsolved. In this study, we focus on a single-layer single-head linear attention model, where an empirical risk minimization is defined for learning. Given a finite number of training examples, we determine the exact solution of network-parameter dynamics, which is linked to the training error and generalization error, characterizing how memorization and generalization emerge from learning and how these transitions are related to curvatures of the empirical risk landscape. Furthermore, we calculate a data-dependent free energy, from which the training and generalization errors in the long time limit of learning dynamics can be well predicted. Our work adds insights toward understanding the mechanism of in-context learning especially from a learning dynamics perspective.

cs.LG · cond-mat.dis-nn

版本更新 · v2arXiv:2604.13511

Phase transition in compressed sensing using log-sum penalty and adaptive smoothing

Keisuke Morita, Federico Ricci-Tersenghi, Masayuki Ohzeki

原文摘要

In many real-world problems, recovering sparse signals from underdetermined linear systems remains a fundamental challenge. Although $\ell_1$ norm minimization is widely used, it suffers from estimation bias that prevents it from reaching the Bayes-optimal reconstruction limit. Nonconvex alternatives, such as the log-sum penalty, have been proposed to promote stronger sparsity. However, maintaining their algorithmic stability is challenging. To address this challenge, we introduce an adaptive smoothing strategy within an approximate message passing framework to mitigate algorithmic instability. Furthermore, we evaluate the typical exact-recovery threshold for Gaussian measurement matrices using the replica method and state evolution. The results indicate that the adaptive method achieves exact recovery over a broader region than $\ell_1$-norm minimization, although metastable states hinder the attainment of the information-theoretic limit.

cs.IT · cond-mat.dis-nn

新论文 · v1arXiv:2611.20301

Multi-output neural network surrogate for double-pulse femtosecond optothermal control of thin silicon films on fused silica

George D. Tsibidis

原文摘要

We introduce a multi-output feedforward neural network surrogate trained on a coupled multiple-reflection and two-temperature model to map double-pulse femtosecond laser excitation (1030 nm) of crystalline silicon films on fused silica across film thicknesses (50-500 nm) and inter-pulse delays (0-10ps). Trained via Levenberg-Marquardt optimization on 2000 physical simulations, the dual-hidden-layer network ([40,30] neurons) simultaneously predicts six strongly coupled optothermal observables, peak electron temperature, peak lattice temperature, peak carrier density, minimum and maximum absorptivity and minimum refractive index achieving coefficients of determination R2 larger than 0.9998 across all targets. Evaluating a dense 200 by 200 design grid in milliseconds resolves pronounced thin-film interference fringes and carrier-induced index modulation, identifying an optimal optothermal coupling regime at d=156 nm and Dt=4.572 ps. This unified machine-learning workflow eliminates computationally expensive multi-physics runtime bottlenecks for real-time laser processing control and inverse photonic thin-film design.

physics.optics · cond-mat.dis-nn · cond-mat.mtrl-sci · cs.AI

新论文 · v1arXiv:2611.19848

A phase diagram for training and inference in language models

Zhuo-Yang Song, Zeyu Li, Qing-Hong Cao, Ming-xing Luo, Hua Xing Zhu

原文摘要

A language model is a complex system whose behavior is determined jointly by the Transformer and the corpus, yet how the representation geometry in the Transformer relates to the actual behavior of language models remains open. We analyze how the representation geometry changes along the forward propagation of actual language models and characterize the representation geometry of a document at a given layer by two collective coordinates, document distinguishability and document dimension. On the phase diagram spanned by these collective coordinates, inference is represented as a flow along depth. We find that, for untrained models, depending on their initialization scale, most of the 156 inference flows move toward the collapse and noise regions of the phase diagram. By contrast, the inference flows of 8 trained models from 5 architecture families all enter and reside in the central region of the phase diagram. Describing the evolution of the flows with a discrete Langevin equation, we find that the difference between the inference flows of untrained runs and those of trained models reduces to whether the effective force on the flows in this region is of saddle or stable-node type. On Pythia-1.4B, we interpret the training process as a phase transition. These results give a unified gateway to the relation between representation geometry and language-model behavior.

cs.LG · cond-mat.dis-nn · cond-mat.stat-mech · cs.CL

新论文 · v1arXiv:2611.18420

Graph diffusion transformer for interpretable generation and inverse design of disordered atomic structures

Jiani Hu, Chenyan Wang, Yongkun Yang, Guanyu Hou, Jihaochen Zhang, Lei Liu, Ji Chen

原文摘要

Atomic disorder offers a potent degree of freedom to tailor material properties, yet the resulting vast configuration space obscures structure-property relations and hinders rational inverse design. Here we introduce Graph Amorphous Diffusion Transformer (GaDiT), a generative model that combines dynamic periodic graphs, masked conditioning and an interval-weighted training-noise distribution. GaDiT generates high-quality disordered structures while accelerating sampling by two orders of magnitude compared with existing corrector-based approaches, as demonstrated across two-dimensional amorphous carbon and three-dimensional amorphous silicon and silica. Moreover, GaDiT supports conditional generation under single or combined structural features, physical properties and preparation conditions in a unified model, and enables inverse design that expands accessible structure-property ranges beyond the training distribution. To interpret these structures and the trained network, we propose a conditioned atomic response mapping analysis, revealing both the global structural descriptors and the local atomic environments that govern target properties. Overall, GaDiT establishes a general framework for interpretable generation, inverse design and structure-property discovery for disordered atomic structures.

cond-mat.mtrl-sci · cond-mat.dis-nn · cond-mat.mes-hall · cs.LG · physics.comp-ph

新论文 · v1arXiv:2611.18248

Dancing Tokens on the Vocabulary Network from LLM Generation

Yuan-Chao Hu

原文摘要

Large language models (LLMs) generate text one token at a time, yet how a particular generation relates to the geometry learned in pre-training remains unclear. The fundamental explainability of LLMs is elusive. Here, we build a similarity network from the input embeddings of six open LLMs and map generation onto it. The token-embedding network is scale-free while heavy-tailed, as are random embeddings with matched covariance, but far more fragmented. Distillation barely changes this topology because input embeddings are largely inherited. The knowledge space is thus deterministic from pre-training. Consecutive generated tokens are rarely graph neighbours and, in most models, no more often than in shuffled order, so generation is not a walk on the predefined network, although the tokens it visits are concentrated within it. Through the logit lens in a layer- wise manner, tokens settle late in depth in most models, and repetitive models remain confined to small domain-specific sub-graphs. These observables could serve as inference-time monitors of degeneration. We thus provide a fresh perspective on understanding LLM generation in the microscopic way from node or link prediction by graph learning.

cs.AI · cond-mat.dis-nn

版本更新 · v2arXiv:2606.28105

随机语言模型的标度极限

Scaling limit of the Random Language Model

Eric De Giuli

AI 速读 基于摘要

作者在隐藏符号数N趋于无穷、语法温度以固定x=ε̃_d log N趋于零的标度极限下,为随机语言模型建立定量理论,并推导出冻结转变及语言统计规律。

cond-mat.dis-nn · cond-mat.stat-mech · cs.CL

查看原文摘要

We develop a quantitative theory of the Random Language Model (RLM), an ensemble of stochastic context-free grammars, in a scaling limit where the number of hidden symbols $N \to \infty$ while the grammar temperature $\tildeε_d \to 0$ at fixed $x = \tildeε_d \log N$. In this limit, the model admits a controlled description based on a large-deviation principle over rule-usage patterns. A semi-annealed approximation maps the problem to a class of Random Energy Models with nontrivial combinatorics. We show that the RLM exhibits a freezing transition at a critical value $x_c=1/8$, below which rule usage condenses on a subset of preferred rules and language statistics acquire a nontrivial dependence on corpus length. A second characteristic scale at $x=1/2$, associated with the breakdown of Gaussian fluctuations, separates transition and babbling regimes. Across regimes, we derive explicit scaling laws for the number of distinct rules, entropy, and energy, identifying distinct frozen and critical regimes controlled by the interplay of grammar size, corpus length, and temperature. The theory resolves previous ambiguities regarding the existence of a thermodynamic transition and explains the slow approach to the large-$N$ limit as a consequence of the dependence on $\log N$. It further provides a unified framework in which universal statistical properties of language emerge from typical realizations of generative grammars, with implications for both natural language statistics and the behavior of large language models. Heaps' law and Zipf's law are both derived, and the dependence of predictive entropy on context length is rationalized. Comparison with empirical measurements of Heaps' and Zipf's laws yields parameter estimates for real languages.

版本更新 · v2arXiv:2605.10613

Exact Fixed-Point Constraints in Neural-ODEs: Injecting Prior Knowledge into the Network Parameters

Feliciano Giuseppe Pacifico, Duccio Fanelli, Lorenzo Buffoni, Lorenzo Chicchi, Diego Febbe, Raffaele Marino

原文摘要

Neural ordinary differential equations (Neural-ODEs) learn a velocity field from data. Often, however, additional information on the target field is available, such as the location of some of its equilibria or, more generally, its values at given points. We introduce a method to inject this information directly into the parameters of a Neural-ODE, so that a finite set of prescribed fixed points holds exactly throughout training, with no penalty terms and no post-hoc projection. The output matrix is parametrized through the Moore-Penrose pseudoinverse, implemented by a thin QR factorization of the hidden features at the prescribed points. This splits the output matrix into a component determined by the constraints and a free one, which shapes the field away from the prescribed points without altering its values there. As a result, the constrained Neural-ODE remains a universal approximator of the continuous vector fields compatible with the constraints. We test the method on a two-dimensional Lotka-Volterra competition model with four equilibria. In vector field regression, the constrained model keeps the fixed-point residual at single-precision round-off, and its test error is up to about five times lower than that of an unconstrained network of the same architecture. A soft penalty on the equilibria lowers the residual only at the cost of accuracy, and still leaves it at least an order of magnitude above that of the constrained model. Trained to convergence on datasets of increasing size, the unconstrained and penalized networks saturate already on small datasets, while the constrained model keeps improving over a wider range. Close to the equilibria and over long time horizons, the constrained model is the most accurate at every dataset size considered. This also holds when the models are trained on observed trajectories, by backpropagation through the ODE solver.

cond-mat.dis-nn · cs.LG · physics.bio-ph

版本更新 · v2arXiv:2605.06367

The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models

Flavio Nicoletti, Chenxiao Ma, Enrico Ventura, Luca Saglietti, Stefano Sarao Mannelli

原文摘要

Real-world datasets are inherently heterogeneous, yet how per-class structural differences and sampling imbalance shape the training dynamics of diffusion models---and potentially exacerbate disparities---remains poorly understood. While models typically transition from an initial phase of generalization to memorizing the training set, existing theory assumes homogeneous data, leaving open how class imbalance and heterogeneity reshape these dynamics. In this work, we develop a high-dimensional analytical framework to study class-dependent learning in score-based diffusion models. Analyzing a random-features model trained on Gaussian mixtures, we derive the feature-covariance spectrum to characterize per-class generalization and memorization times. We reveal the explicit hierarchy governing these dynamics: class variance is the primary determinant of learning order---consistently favoring higher-variance classes---while centroid geometry and sampling imbalance play secondary roles. Specifically, sampling imbalance acts as a modulator that can reverse this ordering and, under strong imbalance, forces minority classes to acquire distinct, delayed inference times during backward diffusion. Together, these results suggest that diffusion models can memorize some classes while others remain insufficiently learned. We validate our theoretical predictions empirically using U-Net models trained on Fashion MNIST.

stat.ML · cond-mat.dis-nn · cs.LG

版本更新 · v3arXiv:2606.03217

An Asymptotic Theory of Chain-of-Thought in In-Context Learning

Kaito Takanami, Cengiz Pehlevan

原文摘要

Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood. To address this question, we study a theoretically solvable model of CoT for in-context weight prediction in linear regression, where test-time reasoning is represented as an iterative refinement of the weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, we derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length. Our analysis reveals a sharp phase transition separating exponential and polynomial improvement, saturation, and overthinking, and characterizes how the optimal reasoning depth scales. We further show that deeper reasoning is most effective with sufficiently rich pretraining and in-context information, whereas limited pretraining or context makes longer reasoning prone to error amplification or saturation. We also validate these predictions through experiments on fully learned linear attention and softmax attention models. Our results provide a unified theoretical account of how test-time CoT depth affects generalization.

stat.ML · cond-mat.dis-nn · cs.LG

新论文 · v1arXiv:2611.14782

Bayesian inference with a polymer probabilistic computer

Travis Wanless, Yuriy Bandera, Lubomir Kostal, Stephen H. Foulger

原文摘要

Bayesian inference weighs competing explanations of noisy data by sampling from a posterior distribution, a task that deterministic hardware performs only by simulating randomness. Probabilistic bits (p-bits) supply this randomness physically, but the few hardware networks of p-bits built so far use inorganic devices, address optimization or learning, and are judged without reference to the spread expected from finite sampling. Here we report a probabilistic computer built from solution-processed conjugated-polymer memristors. Four polymer p-bits, tracked online for drift and coupled in a closed Gibbs-sampling loop, sample competing, degenerate and cooperative posteriors over sixteen states with statistics that fall within the range of ideal p-bits, both in their distribution and in their autocorrelation, without any parameter fitted to the target. This agreement extends to new simulated sensor readings whose outcome was predicted before each run, including readings that point away from the true answer. Reprogrammed in software, the same circuit and controller reproduces the invertible AND gate of earlier p-bit networks, with its temperature fixed in advance rather than fitted. Polymer p-bits thus open a route to printed, large-area probabilistic computers that infer alongside their sensors.

cs.ET · cond-mat.dis-nn · cond-mat.mtrl-sci · cond-mat.soft

新论文 · v1arXiv:2611.14744

Beyond Ground States: Learning Many-Body Hamiltonians for p-Bit Invertible Arithmetic

Tian Feng, Xutang Fan, Yiyun Chen

原文摘要

Probabilistic-bit (p-bit) invertible logic encodes a Boolean relation as an energy whose ground states are its legal assignments, computing in any direction by sampling. Traditional constructions target ground-state membership and a wide minimum energy gap, yet finite-time samplers traverse excited states. We cast Hamiltonian design as inverse statistical mechanics: fit an energy with interactions up to order four whose Gibbs law approaches uniformity on each clamp case's legal states, under a gap floor keeping every legal state below every illegal one. Constrained maximum likelihood with pruning and refitting fits terms from a fixed pool, and every delivered energy is audited exactly on all cases. Each energy is scored by time to a verified solution under restarts with its best of nine tuned samplers. In underdetermined modes, learned multipliers factor in 0.55, 0.44, 0.52 and 0.46 [0.38, 0.56] of the hand-written encodings' updates from 4x4 to 7x7, faster within every family below 7x7 at no resolved work penalty; learned adders solve the reverse mode in 0.71 of the updates at 8 bits and about a quarter at 24 and 32 bits, at 0.55 of the work at 32 bits. At 5x5 under equal pool, budget and floor, a gap-maximizing fit reaches 0.79 [0.73, 0.88] in updates and 1.26 in work against the hand-written encoding, while likelihood fitting takes 0.55 [0.46, 0.58] and 0.58 [0.48, 0.61] of the gap fit's updates and work. Rounding coefficients to the narrowest exact integer words tested, we run both energies at each pair's common word on one FPGA color-sweep SSA core, bit-exact with software: learned multipliers take 0.41 of the hand-written encoding's cycles at 4x4 and 0.057 [0.045, 0.068] at 7x7, at up to 2.4 times its lookup tables. The method requires only a circuit's legal assignments; in determined modes, hand-written encodings take less work, and from 12 bits on the adder fewer updates.

cond-mat.dis-nn · cs.ET · physics.app-ph

新论文 · v1arXiv:2611.14221

基于 Transformer 的语言模型中由内向外计算的阶段性且可解释的涌现

Stagewise and interpretable emergence of inside-outside computation in Transformer-based language models

Jack T. Parley

AI 速读 基于摘要

作者在具有可变解析树拓扑的合成上下文无关文法上研究 Transformer 的学习动力学,报告了可解析计算、可解释的阶段性损失曲线,并可视化由内向外计算随深度与训练过程的涌现。

cs.CL · cond-mat.dis-nn

查看原文摘要

Variability in syntactic structure and parsing ambiguity are fundamental aspects of natural language which modern LLMs have to learn to deal with proficiently. Previous "post-mortem" studies have shown that fully trained Transformers indeed approximate canonical parsing algorithms such as inside-outside, but little is established theoretically about how this is actually learned. Here, we make a first step towards addressing this by studying the learning dynamics in synthetic context-free grammars with variable parse tree topology. The stagewise learning curves are fully interpretable, in the sense that all of the loss stages a) can be computed analytically, and b) can be clearly associated to the representation of a hidden latent tree variable, which allows to lift ambiguity and utilize information from wider contexts. By probing representations, we visualize the emergence of inside-outside computation across depth and during training: while during early stages local (bigram/trigram) information is used and involves only a few initial layers, gradually the Transformer develops an encoder-decoder structure where the most abstract latent information is encoded in the middle, and the latter half of the network is used to progressively refine the final token prediction. This organization of computation is reminiscent of findings in real-world LLMs, and importantly applies here to a synthetic model which allows full analytical control of the data statistics, thus opening the door to theoretical progress.

新论文 · v1arXiv:2611.13146

Everything in its right place: attention is a radio head

Lluís Arola-Fernández, Aissam Khadraoui, Alex Arenas

原文摘要

Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order. We show that attention can instead emerge from the synchronization of a receiver with its context tokens. Each past token is an oscillator whose state carries its content and whose frequency drifts with its age. The present token selects them by content, aligning an antenna with its query, and by position, setting the frequency of a receiver with its dial. Solving Kuramoto's problem for this receiver yields an attention layer in closed form with two phases: above a critical coupling set by the context, the receiver locks and its weights normalize; below it, it stays silent and its weights sum to less than one. Trained networks use both phases to select, abstain and count, fail where the theory predicts and keep reading far beyond the training length. Each part of attention finds its place in one device, a radio head: it tunes in to what to hear and when, and stays silent when nothing relevant is on air.

physics.soc-ph · cond-mat.dis-nn · cs.AI · cs.LG · nlin.AO

版本更新 · v2arXiv:2609.07591

Topology Obstructs Pure Foundation Neural Quantum States

Timothy Heightman, Elena Orlova, Philip Mantrov, Aleksei Ustimenko

原文摘要

Foundation models for ground states in spin-1/2 systems are a promising method for problems ranging from quantum chemistry to identifying new phase diagrams. Nearly all such models are currently pure-states that condition on the Hamiltonian's parameters, whose Monte Carlo samples give energy estimates according to the variational principle. In this contribution, we show that this representation is topologically obstructed. For any gapped Hamiltonian family whose ground-state bundle is non-trivial, every continuous normalized state-vector model has zero fidelity with the ground state at some parameter value in the Hamiltonian family. For that value, the energy is at least one spectral gap, $Δ$, with an $O(Δ)$ gap in an open-neighbourhood of that point. We show that this is a sufficient no-go also in the case of degenerate ground-state manifolds, time dynamics, and periodic systems with mixed space-time topology, demonstrating these obstructions on one- and two-qubit systems. We discuss how this causes a spike in the fidelity susceptibility, giving a numerical signature of a phase-transition where there is none. Finally, we show that operator-valued models canonically avoid these obstructions and preserve topological information, implying a structural necessity in representation for foundation neural quantum states.

quant-ph · cond-mat.dis-nn · cond-mat.str-el · cs.AI · math-ph

版本更新 · v2arXiv:2611.08506

Structure is in the zoom: probing neural symmetry through dimensionality scaling

Adil Amin

原文摘要

Scaling exponents of neural dimensionality are read as properties of the code, and used to compare representations and benchmark models against cortex. Yet one recording of 11,311 mouse V1 neurons gives 0.25, 0.30 or 0.35, depending only on how it is subsampled. We show that such an exponent measures the population and the probe together. We split it exactly into a floor, which random subsets of the same sizes give, and a shift, which a declared axis of stimulus classes adds. The usual random-subsample exponent is almost all floor. Stimulus direction lifts the exponent in every grating recording by reading both symmetry sectors of the code: orientation, which repeats every half turn, and direction, which does not. The lift lives in trial-to-trial fluctuations that differ between directions, while the class means, which linear decoders read, pull it down. Two measured terms, how much the classes' fluctuations overlap and how the means sit against them, account for the shift. They give its sign where it changes: in Allen Neuropixels recordings, whose class means mostly raise the exponent, and in macaque V1. In an XY lattice whose classes stay on one orbit, temperature flips the sign as the two terms say, and in GPT-2 the shift ranks the indirect-object circuit's name-mover and backup heads. Reported with its path, its floor and its shift, an exponent says how much structure a probe finds and what carries it. Our open tool, rung, reports all three for any labelled neural recording or language-model layer: https://github.com/adilamin89/structure-in-the-zoom

q-bio.NC · cond-mat.dis-nn · cs.CL · cs.LG