跳到正文
研踪PaperTrail

论文库

检索计算机领域论文，按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+0新增论文
+0其中 AI 相关
+2已有论文修订
+100新增 AI 导读

arXiv 最近完成扫描：2026-11-15 01:33 · 全站最近入库：2026-11-14 15:09各信源进度 →

统计口径与数据新鲜度

统计 2026-11-14 01:54 — 2026-11-15 01:54（北京时间） · 点击数字查看对应简报；AI 相关包含在新增论文内。

最近完成检查：2026-11-15 01:33最近发现新增版本：2026-11-14 15:09库内论文最新更新时间：2026-11-13 02:59

每分钟刷新已保存的数据。空检查不代表采集异常；新收录可能包含历史论文。AI 导读按新增版本计数。

23,348 已收录论文19,580 AI 相关6,729 有修订112 HF 社区发现

浏览论文库

研究日报 / 周报 →按信源浏览 →

时间范围跟随排序：版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常，查看状态 →

找到 769 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-15 01:32（北京时间）
版本更新 · v3arXiv:2509.21172

Inverse Reinforcement Learning with Just Classification and a Few Regressions

Lars van der Laan, Nathan Kallus, Aurelien Bibaut

原文摘要

Inverse reinforcement learning (IRL) seeks to recover a reward function from observed behavior, but rewards are typically only partially identified: many reward--value pairs can induce the same behavior policy. We study this problem in the maximum-entropy, or Gumbel-shock, model under a broad class of statewise affine normalization constraints, with anchor-action constraints as a special case. This leads to Generalized Policy-to-$Q$-to-Reward (GenPQR), a modular approach to normalized reward recovery based on policy estimation and $Q$-evaluation via the Bellman equation; both components can be instantiated with off-the-shelf classification and regression methods. We establish modular finite-sample guarantees under general function approximation, with separate terms for policy-estimation and $Q$-function estimation error. We also give explicit bounds for GenPQR with fitted $Q$-evaluation. In experiments, GenPQR improves on DeepPQR in reward recovery. Our guarantees extend beyond anchor actions, accommodate large action spaces, and are not tied to a specific neural-network architecture or training procedure.

cs.LG · econ.EM · math.OC · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2604.21691

There Will Be a Scientific Theory of Deep Learning

Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth 等 14 位作者

原文摘要

In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.

stat.ML · cs.LG

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.16097

A Matching Lower Bound for Online Smooth Calibration

Kaixuan Ji, Quanquan Gu

原文摘要

We study smooth calibration error in sequential binary prediction, which measures how far a sequence of probabilistic forecasts is from being calibrated and, unlike the classical expected calibration error, is robust to small perturbations of the forecasts. Against adaptive adversaries, the minimax smooth calibration error is known to lie between $Ω(T^{1/3})$ and $O(\sqrt{T})$ (Qiao and Zheng, 2024), where $T$ is the number of rounds. In this paper, we close this gap by constructing a new adversary whose outcome mean moves with the forecaster's cumulative residual. This design makes compensating for past residuals costly, and forces every forecaster to incur an expected smooth calibration error of $Ω(\sqrt{T})$. This lower bound matches the known upper bound up to universal constants, and further implies $Ω(\sqrt{T})$ lower bounds for calibration distance and lower calibration distance. Overall, our results settle the minimax rates of these calibration measures in sequential prediction. The proof ideas are partially developed by GPT-6 Astra under the authors' guidance.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.16039

Improved Elastic Weight Consolidation as an Optimization Constraint for Continual Learning

Davide Wiest

原文摘要

Elastic Weight Consolidation (EWC) is a widely used regularization-based technique for continual learning, protecting performance on old tasks with a quadratic penalty whose empirical Fisher is usually justified through probabilistic curvature. We instead introduce Continual Learning as an Optimization Constraint (CLOC): new training must preserve chosen functionals of learned behavior. EWC's empirical Fisher is exactly the first-order form of this constraint when the preserved functional is each sample's loss. This derivation exposes a flaw: EWC implicitly weights each sample by the scale of its loss gradient, so well-fitted samples lose importance. EWC-DR observed this failure in classification; CLOC identifies its general cause. Improved Elastic Weight Consolidation (IEWC) penalizes output changes according to the first-order approximation of each old sample's loss level set, removing the weighting for any differentiable loss and output geometry, and prevents corrupt samples from dominating the importance matrix. Across classification, segmentation, language, forecasting, and diffusion, IEWC variants match or improve on EWC and outperform EWC-DR. In a held-out evaluation with validation-selected hyperparameters, uniform IEWC improves ImageNet-R accuracy over matched EWC by 2.00 percentage points (95% CI 0.93-3.07).

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15990

Characterizing the Edge of Stability in Variational Training Without Priors

Juraj Marusic, Jonathan Wenger, Beau Coker, John P. Cunningham

原文摘要

Variational approaches in deep learning promise to deliver improved uncertainty quantification and out-of-distribution generalization by inferring a distribution over weights, regularized by a divergence to a chosen prior. However, prior elicitation presents a significant challenge in deep learning, resulting in the common practice of significantly downweighting the regularization term in the variational objective. Recently, it was proposed to train solely via the expected loss by relying on implicit regularization from the optimizer, rather than explicit regularization from the divergence to the prior. While this approach largely sidesteps the issue of prior elicitation, current theory lacks an understanding of how key optimization hyperparameters impact the implicit regularization. Here, we theoretically and empirically demonstrate that larger learning rates and fewer parameter samples lead to flatter minima of the loss landscape, which are known to generalize better. We show that training on the expected loss operates at a modified edge of stability, characterized by the signal-to-noise ratio of the gradient estimates and the number of Monte Carlo samples. Experiments on out-of-distribution benchmark datasets confirm the practical significance of the theoretical results.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2602.08151

A second order regret bound for NormalHedge

Yoav Freund, Nicholas J. A. Harvey, Victor S. Portella, Yabing Qi, Yu-Xiang Wang

原文摘要

We consider the problem of prediction with expert advice for "easy" sequences. We show that a variant of NormalHedge enjoys a second-order $ε$-quantile regret bound of $O(\sqrt{V_T \log(V_T/ε)})$ when $V_T > \log N$, where $V_T$ is the cumulative second moment of the instantaneous per-expert regret, averaged with respect to a natural distribution $\mathbf{q}_t$ determined by the algorithm. The algorithm is motivated by a continuous time limit using Stochastic Differential Equations. The discrete time analysis uses self-concordance techniques.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15829

Training Calibrated Conditional Generative Models with a Sample-Only Coverage Loss

Benjamin Remy, Raphael Rossellini, Chihway Chang, Rebecca Willett

原文摘要

Conditional generative models have become the tool of choice for sampling posteriors in scientific inverse problems, where we seek to sample images or signals conditioned on a set of indirect and noisy measurements. Yet they are often overconfident, which is a critical failure in scientific applications where accurate uncertainty quantification is essential. Existing regularizers fall short in complementary ways: differentiable coverage objectives require the model's density and do not identify the posterior, while strictly proper scoring rules such as the energy score need only samples but do not directly enforce coverage. We close this gap by introducing a differentiable loss function based on the expected coverage probability, estimated from samples alone. It applies to any conditional generative model, with no need for tractable densities or architectural changes. We prove that it is a strict divergence, vanishing if and only if the model equals the true posterior. On simulation-based inference benchmarks, it matches a density-based coverage loss without evaluating the density. On high-dimensional imaging inverse problems, it yields better-calibrated posteriors than scoring-rule regularizers, with little change in the reconstruction quality of point estimates.

cs.LG · stat.ME · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2512.10817

Extrapolation of Periodic Functions Using Binary Encoding of Continuous Numerical Values

Brian P. Powell, Jordan A. Caraballo-Vega, Mark L. Carroll, Thomas Maxwell, Andrew Ptak, Greg Olmschenk, Jorge Martinez-Palomera

原文摘要

We introduce Normalized Base-2 Encoding (NB2E) as a method for encoding continuous numerical values and demonstrate that, using this input encoding, vanilla multi-layer perceptrons (MLP) successfully extrapolate four periodic waveforms without prior knowledge of their functional form. We contrast this finding with fixed Fourier encoding, which extrapolates well when the number of signal cycles over the normalized domain is an even integer and loses coherence at most other values, whereas NB2E degrades gradually away from those values and, except near odd cycle counts, remains substantially more accurate. These findings are limited to a single MLP architecture trained on one synthetic, exactly periodic, one-dimensional signal at a time, with extrapolation extending only to the end of the normalized domain. We do not propose a mechanism for this phenomenon. Rather, we present empirical observations which merit further study.

cs.LG · cs.AI · cs.CV · stat.ML

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15816

Global-Local and Window-Concatenated Functional Random Forests: Early and Late Fusion of Window-Specific Functional Principal Components

Fabrizio Maturo, Annamaria Porreca

原文摘要

Functional random forests built on a single functional principal component decomposition of the whole domain may dilute features confined to short intervals, whose location is rarely known in advance. We propose two random forests based on window-specific functional principal components. The global-local functional random forest (GL-FRF) is a late-fusion method that stacks a tuned full-curve forest and one forest per window with penalised convex weights; it contains the full-curve forest as a feasible solution, and its weights indicate which parts of the domain drive the prediction. The window-concatenated functional random forest (WC-FRF) is an early-fusion method that trains a single tuned forest on the concatenated window-specific scores. On twenty UCR datasets, GL-FRF improved on a tuned full-curve forest with the same number of trees on 16, with Brier score reductions of up to 0.21, significantly after Holm adjustment. No method was best on more than six datasets. WC-FRF was more accurate than GL-FRF on 13 datasets but failed when the signal required the full curve, whereas GL-FRF had the smallest mean and maximum excess over the best method across all 32 simulated and real settings, which makes it the safer choice when the location of the signal is unknown.

stat.ML · cs.LG · stat.ME

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15778

From Link Prediction to Linear Community Detection: A Finite-Sample Guarantee for Graph Pretraining

Ying Chen, Zhangyang Wang, Yixuan He

原文摘要

Link prediction (LP) is a standard pretraining objective for graph neural networks (GNNs): hide edges, train an encoder-decoder to predict them, and reuse the node embeddings for downstream node tasks. Despite its empirical success, there is little quantitative theory explaining why an edge-level objective should produce embeddings that are linearly useful for node classification. We study this question in the finite-sample stochastic block model (SBM). For any bounded GNN encoder paired with a symmetric bilinear decoder, we prove a conditional reduction: small population held-out LP excess risk implies small community-classification error for a transductive linear classifier on the learned embeddings. The bound decomposes into an optimization residual, an architectural approximation residual, and explicit statistical terms of order $\tilde{O}(\sqrt{d/n})$ in bounded dense regimes. We further give a constructive few-label classifier: nearest-centroid classification under the positive semidefinite quadratic form $G = W Z^\top Z W$, whose label error is coupled to LP quality. The framework is architecture-agnostic through the approximation residual, and we instantiate this residual as $O(\log n / n)$ for a graph isomorphism network (GIN) with spectral positional encoding and a bilinear decoder. Synthetic SBM experiments validate the theorem chain from LP risk to score error to classification error, and show when the worst-case constants become conservative.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2605.06367

The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models

Flavio Nicoletti, Chenxiao Ma, Enrico Ventura, Luca Saglietti, Stefano Sarao Mannelli

原文摘要

Real-world datasets are inherently heterogeneous, yet how per-class structural differences and sampling imbalance shape the training dynamics of diffusion models---and potentially exacerbate disparities---remains poorly understood. While models typically transition from an initial phase of generalization to memorizing the training set, existing theory assumes homogeneous data, leaving open how class imbalance and heterogeneity reshape these dynamics. In this work, we develop a high-dimensional analytical framework to study class-dependent learning in score-based diffusion models. Analyzing a random-features model trained on Gaussian mixtures, we derive the feature-covariance spectrum to characterize per-class generalization and memorization times. We reveal the explicit hierarchy governing these dynamics: class variance is the primary determinant of learning order---consistently favoring higher-variance classes---while centroid geometry and sampling imbalance play secondary roles. Specifically, sampling imbalance acts as a modulator that can reverse this ordering and, under strong imbalance, forces minority classes to acquire distinct, delayed inference times during backward diffusion. Together, these results suggest that diffusion models can memorize some classes while others remain insufficiently learned. We validate our theoretical predictions empirically using U-Net models trained on Fashion MNIST.

stat.ML · cond-mat.dis-nn · cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2610.03951

Evolving LLM-Generated Features for Interpretable Classification

Jack Butler, Zainab Afolabi, Nikita Kozodoi

原文摘要

Large language models (LLMs) are increasingly used as classifiers, yet they operate as opaque systems whose decisions are difficult to interpret, which complicates their use in regulated domains such as credit scoring or medical diagnosis. We propose an evolutionary framework that iteratively discovers natural language feature definitions (rubrics) for interpretable classification. An LLM generates candidate binary features, evaluates each sample against them, and the resulting vectors can be used to train a transparent classifier such as logistic regression. The feature set evolves over multiple iterations guided by classification errors, per-class activation rates, and feature ablation scores. We evaluate across three benchmarks, including a credit risk dataset representative of regulated domains, comparing single-shot LLM rubrics, evolved rubrics, and direct zero-shot LLM classification. Evolved features improve over single-shot rubrics by +2.9 pp on average and outperform zero-shot LLM classification on two of three tasks, while providing fully auditable decision logic. On the credit risk task, the zero-shot LLM performs at chance (50.7%) with a strong bias toward a single class, whereas evolved features achieve balanced, interpretable predictions. Crucially, this failure is invisible in aggregate accuracy and surfaces only under per-class auditing. Analysis reveals that evolution is most effective when label boundaries cannot be inferred from category names alone or when the LLM lacks reliable domain-specific reasoning.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v3arXiv:2507.21135

Quantum Geometry of Data

Alexander G. Abanov, Luca Candelori, Harold C. Steinacker, Martin T. Wells, Jerome R. Busemeyer, Cameron J. Hogan, Vahagn Kirakosyan, Nicola Marzari 等 12 位作者

原文摘要

We demonstrate how Quantum Cognition Machine Learning (QCML) encodes data as quantum geometry. In QCML, features of the data are represented by learned Hermitian matrices, and data points are mapped to states in Hilbert space. The quantum geometry description endows the dataset with rich geometric and topological structure---including intrinsic dimension, quantum metric, and Berry curvature---derived directly from the data. QCML captures global properties of data through a compact operator representation. At fixed Hilbert-space dimension, the number of operator parameters grows only linearly with the number of features. The present work provides the first full exposition of QCML as a framework for matrix geometry, establishing its mathematical foundation and demonstrating how operator-level tools from quantum geometry can be directly applied to data. We introduce the matrix Laplacian and its eigenmaps as a geometry-preserving alternative to graph-based embeddings, and we show how topological invariants such as Chern numbers can be extracted from data for the first time. An important practical outcome is the efficiency of the representation: the relevant geometric structure of high-dimensional datasets can be captured with Hilbert spaces of size as small as $N \sim 10$. We illustrate these results on a number of synthetic and real-world examples, including the Wisconsin Breast Cancer dataset, where QCML uncovers geometric separability between benign and malignant samples and yields an interpretable decomposition of feature relevance. The emphasis of this work is on a new geometric representation of data rather than on algorithmic benchmarking, establishing a foundation for extracting global geometric and topological structure from high-dimensional datasets.

cs.LG · quant-ph · stat.ML

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15698

Learning Mixtures of Exponential Family

Mansi Sood, Devavrat Shah

原文摘要

Motivated by causal inference and physical computing systems, we consider the problem of learning mixture of two bounded-support exponential-family distributions with a shared interaction parameter matrix and mixture component-specific linear parameter vectors. The unknown sequence of component labels belongs to a known structured class. We propose an estimator based on a convex loss that avoids the log-partition function; for each fixed label assignment, estimation is a convex optimization problem. We show that the conditional population loss uniquely identifies the interaction matrix and the linear parameter attached to each sample. For a global empirical minimizer, we establish finite-sample bounds on the interaction error in Frobenius norm and the average squared linear-parameter error. The bounds depend on the logarithm of the number of admissible label assignments, and, for fixed dimension and model bounds, imply consistency when this complexity grows sublinearly with sample size. Additional occupancy and separation conditions yield component and label recovery.

stat.ML · cs.LG · eess.SP

查看论文
arXiv ↗PDF ↗
版本更新 · v3arXiv:2605.22010

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

Margalit Glasgow, Joan Bruna

原文摘要

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network $f_{\hatρ_t^m}$ to its infinite-width counterpart $f_{ρ_t^{MF}}$, which evolves in the mean-field dynamics. While constant-time horizon bounds for $\|f_{ρ_t^{MF}} - f_{\hatρ_t^m}\|$ may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by $L_t$ the mean-field excess MSE loss at time $t$ and $m$ the number of neurons, under standard regularity assumptions and the condition $\int_0^\infty L_t^{1/2} dt =O(\log d)$, we obtain the uniform in time bound $\|f_{ρ_t^{MF}}- f_{\hatρ_t^m}\|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)}$ whenever $L_t \lesssim t^{-c}$. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than $t^{-2}$, we can attain a loss of $ε$ with only $\text{poly}(d/ε)$ neurons, training samples, and GD steps.

stat.ML · cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v3arXiv:2602.03999

Functional Stochastic Localization

Anming Gu, Bobby Shi, Kevin Tian

原文摘要

Eldan's stochastic localization is a probabilistic construction that has proved instrumental to modern breakthroughs in high-dimensional geometry and the design of sampling algorithms. Motivated by sampling under non-Euclidean geometries and the mirror descent algorithm in optimization, we develop a functional generalization of Eldan's process that replaces Gaussian regularization with regularization by any positive integer multiple of a log-Laplace transform. We further give a mixing time bound on the Markov chain induced by our localization process, which holds if our target distribution satisfies a functional Poincaré inequality. Finally, we apply our framework to differentially private convex optimization in $\ell_p$ norms for $p \in [1, 2)$, where we improve state-of-the-art query complexities in a zeroth-order model.

math.PR · cs.DS · cs.LG · math.ST · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2611.12491

Out-of-the-Box Tabular Foundation Models for Bayesian Optimization in Materials Discovery

Klemens Flöge, Simon Bing, Leo Grinsztajn, Eliott Kalfon, Lennart Purucker, Noah Hollmann, Frank Hutter

原文摘要

Bayesian optimization is widely used in active learning loops for materials science. Gaussian processes serve as the de-facto surrogate model family, but require task-dependent kernel choices and can lack the modeling capacity for complex relationships. Tabular foundation models (TFMs) have recently emerged as the state-of-the-art for predictive tasks on structured data, without requiring bespoke kernels or hyperparameter tuning. We evaluate two TFMs, TabPFN and TabICL, as out-of-the-box surrogate models on two standard materials science benchmarks, comparing against a large field of baselines encompassing Gaussian processes, random forests, and CatBoost. Our findings indicate that TFMs are highly competitive surrogate models across the two benchmarks, without task-specific training for active learning or GP kernel selection. Our results motivate developing TFMs further as surrogates for active learning in materials discovery.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15607

FADO: Learning Causal Explanations with Prior-Fitted Networks

Seth Flaxman, Jess Carr, Annabel Jakob, Marek Masiak, Dino Sejdinovic

原文摘要

In regression and classification tasks with tabular datasets, most of the literature on explanation in ML (e.g. SHAP, permutation importance) has focused on predictive attribution: does feature $X_i$ help predict outcome $Y$? At its heart, predictive attribution captures correlation, not causation. We instead ask a causal question: would manipulating feature $X_i$ change outcome $Y$? We define a natural quantity to answer this question, the feature intervention effect: $Δ_i = E[Y\mid do(X_i=+1)]-E[Y\mid do(X_i=-1)]$. In general, estimating this quantity from observational data alone would entail strong causal assumptions. We propose FADO (Feature-wise Amortized Do-Operators), an amortized inference method using prior-fitted networks trained on synthetic tabular datasets generated by structural causal models. During training, the synthetic data-generating process is known, so we can compute ground-truth intervention effects $Δ_i$, and train a model to predict them at test time from a purely observational tabular dataset. As we demonstrate through extensive experiments, our new causal explainability method is easy to train and provides better explanations under a range of settings for which standard predictive attribution fails, including proxy variables, confounders, colliders, and placebos. We evaluate it on real, synthetic and semi-synthetic datasets, with and without hidden confounders, and compare it to existing predictive attribution methods and state-of-the-art causal inference methods.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v3arXiv:2606.03217

An Asymptotic Theory of Chain-of-Thought in In-Context Learning

Kaito Takanami, Cengiz Pehlevan

原文摘要

Chain-of-thought (CoT) reasoning has become a widely used mechanism for eliciting multi-step reasoning in large language models by generating intermediate reasoning steps at inference time. Yet the scaling behavior of generalization with CoT depth remains poorly understood. To address this question, we study a theoretically solvable model of CoT for in-context weight prediction in linear regression, where test-time reasoning is represented as an iterative refinement of the weight-parameter estimate. Using tools from random matrix theory under high-dimensional asymptotics, we derive an exact formula for the generalization error as a function of reasoning depth, pretraining data amount, and context length. Our analysis reveals a sharp phase transition separating exponential and polynomial improvement, saturation, and overthinking, and characterizes how the optimal reasoning depth scales. We further show that deeper reasoning is most effective with sufficiently rich pretraining and in-context information, whereas limited pretraining or context makes longer reasoning prone to error amplification or saturation. We also validate these predictions through experiments on fully learned linear attention and softmax attention models. Our results provide a unified theoretical account of how test-time CoT depth affects generalization.

stat.ML · cond-mat.dis-nn · cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v3arXiv:2610.28910

Backtracking Direct, Indirect and Total Effects

Jose M. Peña

原文摘要

Natural direct and indirect effects are defined through cross-world interventional counterfactuals. We propose an alternative construction based on backtracking counterfactuals. Rather than intervening on the treatment or mediator and change the structural equations, we use a backtracking conditional distribution to construct an alternative exogenous state satisfying a desired treatment-mediator constraint and then evaluate the unchanged structural equations on that state. This yields the backtracking direct and indirect effects, and also the backtracking total effect. Likewise, we adapt the population intervention effects.

stat.ME · cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2605.07665

Debiased Counterfactual Generation via Flow Matching from Observations

Hugh Dance, Johnny Xi, Peter Orbanz, Benjamin Bloem-Reddy

原文摘要

Estimating the full counterfactual distribution under an intervention is central to treatment risk assessment and counterfactual generation tasks. Existing approaches model the counterfactual distribution as a standalone generative target, without exploiting its relationship to the observational data. In this work, we show that under standard assumptions, observational distributions and marginal counterfactual distributions over outcomes are tightly linked: they have identical support and tail behavior, remain statistically close under weak confounding, and share any outcome features which are invariant to confounders. These properties motivate learning counterfactual distributions not from scratch, but via a deconfounding flow from the observational distribution. We formulate this problem via flow-matching and derive a semiparametrically efficient estimator based on a novel efficient influence function correction. We subsequently extend our estimator to target minimal-energy flows in high-dimensions, which we show can be especially simple targets between observational and counterfactual distributions. In experiments, deconfounding flows outperform existing debiased counterfactual distribution estimators, while also mitigating known failure modes of flow-based methods.

stat.ML · cs.LG

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15456

Bet Imaginatively, not Historically in Independent-Data Sequential Testing

Nathaniel Xu, Feng Liu, Danica J. Sutherland

原文摘要

Sequential testing avoids some of the many controversies and drawbacks related to traditional batch testing with p-values. In particular, the testing by betting paradigm has developed rapidly in recent years; it frames the level of evidence against the null as the wealth of a player in a betting game. Naturally, how much a player should bet is a key consideration, and should reflect their confidence in what they know about the next observed data point. Much current work uses standard betting strategies developed for adversarial problems. In the stochastic setting, where the observed data is independent and stationary, these approaches may be suboptimal. We propose a simple betting strategy, follow the leave-one-out e-power (FLOE), that exploits the i.i.d. nature of a data stream to estimate the e-power of our current test on the underlying distribution. We average not over the historical evidence collected so far, as generic online learning-based methods do, but instead over the equally-likely possible data sequences we might have seen based on shuffling the data. With an additional principled regularization technique, this simple drop-in method yields substantial improvements in testing performance on a variety of kernel two-sample and independence testing problems.

stat.ML · cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2603.01514

Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning

Gautam Goel, Mahdi Soltanolkotabi, Peter Bartlett

原文摘要

We study the training dynamics of gradient descent in a multilayer softmax self-attention network trained to perform linear regression and show that in the infinite-data limit this regression problem is equivalent to a matrix factorization problem. This connection motivates the design of Hera, a new first-order optimization algorithm for training self-attention networks. Key features of Hera include a reparameterization of the model arising from the matrix factorization geometry and a rescaling of the parameters which ensures that the empirical loss gradients remain close to the auxiliary matrix factorization gradients throughout training. Hera updates in the reparameterized model are equivalent to a certain preconditioned update in the original parameterization. We also propose a variant of Hera called Hera-PSD; our main theoretical result is that under small Gaussian initialization the excess risk of the model trained by Hera-PSD decreases exponentially fast in the number of gradient descent iterations, up to an additional error term that decreases as $1/n$, up to logarithmic factors, in the size of the training set. A particularly appealing aspect of our excess risk bound is that it does not degrade as the number of self-attention layers increases. In a stylized time-series prediction experiment in a nonlinear dynamical system, we find that a model trained by Hera achieves $98.1\%$ lower mean-squared test error than a model trained by AdamW and $97.7\%$ lower than a model trained by Muon at the same iteration budget.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2508.14821

The C-index Multiverse

Begoña B. Sierra, Colin McLean, Peter S. Hall, Catalina A. Vallejos

原文摘要

Quantifying out-of-sample discrimination performance for time-to-event outcomes is a fundamental step for model evaluation and selection in the context of predictive modelling. The concordance index, or C-index, is a widely used metric for this purpose, particularly with the growing development of machine learning methods. Beyond differences between proposed C-index estimators (e.g. Harrell's, Uno's and Antolini's), we demonstrate the existence of a C-index multiverse among available R and python software, where seemingly equal implementations can yield different results. This can undermine reproducibility and complicate fair comparisons across models and studies. Key variation sources include tie handling and adjustment to censoring. Additionally, the absence of a standardised approach to summarise risk from survival distributions, result in another source of variation dependent on input types. We demonstrate the consequences of the C-index multiverse when quantifying predictive performance for several survival models (from Cox proportional hazards to recent deep learning approaches) on publicly available breast cancer data, and semi-synthetic examples. Our work emphasises the need for better reporting to improve transparency and reproducibility. This article aims to be a useful guideline, helping analysts when navigating the multiverse, providing unified documentation and highlighting potential pitfalls of existing software. All code is publicly available at: www.github.com/BBolosSierra/CindexMultiverse.

stat.ML · cs.LG · stat.AP

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15300

Sharp Covering-Number Bounds for Feedforward ReLU Networks via Parameter Symmetries

Ruizhi Deng, Guohao Shen, Tangrui Xiong

原文摘要

In neural networks, distinct parameter configurations often realize the same input--output mapping, indicating that nominal parameter counts can overstate functional complexity. In this paper, we characterize the parameter-space symmetries of ReLU and Leaky ReLU networks across fully connected, convolutional, and residual architectures. We prove that every quasianalytic parameter map preserving the network function is a composition of positive channel scaling and channel permutation. By quotienting out both symmetries, we derive uniform covering-number upper bounds in which positive scaling reduces the complexity exponent from the total parameter count $S$ to $S-H$ (with $H$ denoting the number of scalable channels) and permutation symmetry divides the bound by the factorial count $G$ of channel permutations. We establish matching lower bounds showing that the exponent $S-H$ is sharp in general and that the divisor $G$ is attained for one-hidden-layer and generically identifiable networks. We further extend these results to general continuous piecewise-linear activated networks. Finally, we apply our covering-number bounds to derive statistical risk guarantees for empirical risk minimization and capacity control beyond i.i.d.\ settings, and show how the underlying symmetries help explain weight rebalancing in function approximation and flat directions in empirical risk optimization.

cs.LG · math.ST · stat.ML

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15270

Communication-Efficient Federated Multi-View Clustering via View-Aware Quantization

Khadija Ardif, Meryeme Ayache, Nidal Nasser, Ouiame Marnissi

原文摘要

Federated multi-view clustering (FedMVC) has recently attracted increasing attention for its ability to exploit complementary clustering structures across multiple clients while preserving data privacy. However, existing FedMVC methods often suffer from high communication overhead due to the repeated transmission of high-dimensional representations, which limits their scalability in practical federated systems. Moreover, different views may contribute unequally to the clustering performance, calling for adaptive communication strategies. In this paper, we propose a view-aware quantization algorithm for communication-efficient FedMVC. By dynamically assigning quantization bits based on iteratively updated view importance scores, the proposed algorithm effectively balances clustering performance and communication efficiency under a global bit budget constraint. To validate its generality, we integrate our approach into four state-of-the-art FedMVC methods. Extensive experiments on multiple benchmark datasets demonstrate that view-aware quantization achieves up to 6.59x communication reduction while maintaining and in some cases surpassing baseline clustering performance, confirming the effectiveness and scalability of the proposed approach.

stat.ML · cs.LG

查看论文
arXiv ↗PDF ↗
新论文 · v1arXiv:2611.15232

Winter wheat improves soil carbon sequestration with reduced gains in warmer climates

Georgios Giannarakis, Vasileios Sitokonstantinou, Dimitrios Bormpoudakis, Ilias Tsoumas, Nikiforos Samarinas, Gustau Camps-Valls, Charalampos Kontoes

原文摘要

Sustainable agricultural intensification requires a precise understanding of how land use decisions influence soil organic carbon (SOC) sequestration under varying environmental conditions. We apply causal machine learning to evaluate the heterogeneous impact of winter wheat-based rotations on SOC at the field-level across Lithuania. Using a nationwide dataset collected over five years (2018-2022), including crop types, management practices, soil properties, and environmental conditions, we estimate how the effects of winter wheat-based rotation vary across space and climate. Our results indicate that winter wheat generally enhances SOC levels, but its effectiveness is modulated by climate. Cooler regions experience higher SOC gains, whereas warmer regions show diminished or even negative effects. Future climate projections suggest that the potential for winter wheat carbon sequestration may decline under high-emission scenarios. These findings highlight the importance of localized agricultural strategies that account for climate variability to maximize SOC sequestration and ensure long-term soil sustainability.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2610.41405

Conditional Neural SDEs for Financial Time-Series Generation

Sina Molavipour, Alireza M. Javid, Björn Löfdahl, Mikhail Nechaev

原文摘要

Conditional generation of time-series paths is a central problem in probabilistic modeling, where the goal is to simulate future trajectories that are both realistic and responsive to external context. It has direct applications in finance and economics. Neural stochastic differential equations provide a natural framework for this task by combining continuous-time stochastic dynamics with learnable drift and diffusion functions. However, directly modeling the target state can be unstable in conditional settings, especially when the target and conditioning variables have strong level dependence, limited training data, or different dimensions. We propose a residual-anchor Neural SDE architecture that decomposes the target into a learned conditional anchor and a stochastic residual. The anchor captures the context-dependent baseline, while the Neural SDE (Stochastic Differential Equation) models the residual dynamics around this baseline. This separation reduces the burden on the stochastic dynamics and generalizes spread-based representations to more general conditioning settings. We evaluate the approach on real-world financial time series, including conditional interest-rate scenarios and a comparison with a time-series foundation model. The results show that the proposed model can generate plausible conditional paths, maintain competitive point accuracy, and provide calibrated uncertainty estimates.

stat.ML · cs.LG

查看论文
arXiv ↗PDF ↗
版本更新 · v8arXiv:2602.07618

Neural Networks With Dense Computational Graphs Are Not Universal Approximators

Levi Rauchwerger, Stefanie Jegelka, Ron Levie

原文摘要

We investigate the approximation capabilities of dense neural networks. While universal approximation theorems establish that sufficiently large architectures can approximate arbitrary continuous functions when there are no restrictions on the weights, we show that dense neural networks do not possess this universality. Our argument is based on a model compression approach, combining the weak regularity lemma with an interpretation of feedforward networks as message passing graph neural networks. We consider ReLU neural networks subject to natural constraints on weights and input and output dimensions, which model a notion of dense connectivity. Within this setting, we demonstrate the existence of Lipschitz continuous functions that cannot be approximated by such networks. This highlights intrinsic limitations of neural networks with dense layers and motivates the use of sparse connectivity as a necessary ingredient for achieving true universality.

cs.LG · stat.ML

查看论文
arXiv ↗PDF ↗
版本更新 · v2arXiv:2611.13219

MediPFN: Amortized In-Context Nuisance Learning and Inference for Causal Mediation Analysis

Shi Bo, Jiaming Hu

原文摘要

Causal mediation analysis decomposes a treatment effect into indirect pathways through mediators and direct pathways through other processes. Tabular foundation models perform classification and regression through in-context learning, providing estimates of conditional means and propensity scores used in causal mediation analysis. Constructing confidence intervals from these estimates requires accounting for uncertainty introduced by fitting and combining several nuisance functions. We introduce MediPFN, an amortized inference framework that adapts a pretrained tabular foundation model to nuisance prediction and combines its predictions through a cross-fitted influence-function estimator. A separate head learns quantiles of the estimator's standardized error from simulated tables with known indirect effects. Supervised fine-tuning, symmetrization, and independent calibration then produce a confidence interval. We establish coverage bounds for the resulting intervals under standard assumptions. Across experiments, MediPFN attains coverage closer to the nominal level than several comparators. It also reduces the computational cost of uncertainty quantification, running 59.1 times faster than bootstrap in matched comparisons.

stat.ML · cs.LG · stat.ME

查看论文
arXiv ↗PDF ↗