跳到正文
研踪PaperTrail

论文库

检索计算机领域论文,按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+1111新增论文
+874其中 AI 相关
+133已有论文修订
+97新增 AI 导读

arXiv 最近完成扫描:2026-11-20 06:44 · 全站最近入库:2026-11-19 19:09各信源进度 →

统计口径与数据新鲜度

统计 2026-11-19 07:09 — 2026-11-20 07:09(北京时间) · 点击数字查看对应简报;AI 相关包含在新增论文内。

最近完成检查:2026-11-20 06:44最近发现新增版本:2026-11-19 19:09库内论文最新更新时间:2026-11-19 02:59

每分钟刷新已保存的数据。空检查不代表采集异常;新收录可能包含历史论文。AI 导读按新增版本计数。

27,982 已收录论文23,249 AI 相关8,559 有修订112 HF 社区发现

浏览论文库

时间范围跟随排序:版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常,查看状态 →

找到 955 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-20 06:42(北京时间)
版本更新 · v3arXiv:2511.07365

Private Sparse Sketches for Linear Regression

Shrutimoy Das, Anirban Dasgupta

原文摘要

Sketching compresses a large dataset into a smaller summary, allowing regression on the sketch to approximate the full-data objective. A sketch released under differential privacy can be reused with any non-private regression solver, any number of times, without further privacy loss. We study private sparse sketches for $\ell_2$ and $\ell_1$ regression. We first prove lower bounds on the noise required by common sparse sketches for both problems when the sketching matrix is public. Since regression on a sketch can be solved without revealing the sketching matrix, we construct sketches whose randomness is kept secret. Hiding how individual records enter the sketch provides an additional source of privacy, allowing less noise for the same privacy guarantee. We prove differential privacy guarantees for both constructions, and experiments on real datasets show improved utility over public-sketch baselines.

cs.LG · stat.ML

版本更新 · v2arXiv:2609.16262

Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent

Alex Buna, Fanghui Liu, Patrick Rebeschini

原文摘要

Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objective reduces the compute available for downstream adaptation. Despite its practical importance, this trade-off is not yet well understood theoretically, even in simple models. In this paper, we cast this allocation as a compute-split problem under a two-stage pretrain--fine-tune procedure with fixed total optimisation budget, using regularised least squares trained by gradient descent as a tractable setting. We characterise the optimal split under data-dependent evaluation geometries induced by the fine-tuning problem. Our results show that the allocation depends on how pretraining directions affect fine-tuning predictions and how fine-tuning shifts are seen through downstream data geometry. In particular, the relevant quantities are determined by prediction-relevant spectral components of the pretraining and fine-tuning empirical covariances. Technically, the analysis relies on a basis-invariant, eigenspace-level spectral decomposition, together with perturbative control of the non-commuting pretraining and fine-tuning dynamics.

stat.ML · cs.LG

版本更新 · v2arXiv:2608.06250

Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification

Alex Buna, Shirley Xiaoqi Liu, Patrick Rebeschini

原文摘要

In overparameterised classification, training data can be linearly separable even when the underlying distribution is not. In this setting, gradient descent (GD) on the logistic loss diverges in norm while converging in direction to a max-margin interpolating classifier, whose implicit bias can be statistically suboptimal. In this work, we show that early stopping can overcome this suboptimality: in a Gaussian mixture model with label-flipping noise, GD stopped at an appropriate oracle time achieves minimax-optimal excess zero-one risk for covariance spectra with fast and continuous decay, including polynomial and exponential spectral decays. Our analysis combines a sharp upper bound for the early-stopped iterate with a matching statistical lower bound over arbitrary classifiers, yielding optimal rates that are validated by experiments. A central technical contribution is a new calibration result that converts excess logistic risk into excess zero-one risk; it handles the model misspecification induced by the label-flipping noise, and removes the square-root rate in standard bounds. We also establish a lower bound for linear interpolators, showing that interpolation can require exponentially more samples than early stopping to achieve the same excess risk.

stat.ML · cs.LG

版本更新 · v2arXiv:2609.14802

From matrix inversion to constraints: provably tighter confidence regions for importance weights in label shift

Mushan Li, Kihyun Han, Yanyuan Ma

原文摘要

Importance weights are essential in domain adaptation under label shift, yet their utility is often undermined by the finite sample uncertainty associated with their estimation. Existing methods typically analyze this uncertainty through Gaussian elimination on interval-valued linear systems, which leads to overly conservative confidence regions and inefficient downstream applications. We propose a paradigm shift from inversion-based inference to a direct matrix constraint framework. We use this framework to define a joint confidence region and extract marginal intervals via linear programming, deriving provably tighter bounds for importance weights while maintaining exact finite-sample validity. Furthermore, we analyze the confidence region's geometry and provide the theoretical results for its diameter bounds. Evaluated across text, image, multimodal benchmarks, including AGNews, MNIST, CIFAR-10, N24News, and a real-world autonomous driving dataset, nuImages, our approach consistently yields shorter confidence intervals and smaller prediction sets than inversion-based methods.

stat.ML · cs.LG

版本更新 · v2arXiv:2510.00569

Guaranteed Noisy CP Tensor Recovery via Riemannian Optimization on the Segre Manifold

Ke Xu, Yuefeng Han

原文摘要

Recovering a low-CP-rank tensor from noisy linear measurements is a central challenge in high-dimensional data analysis, with applications spanning tensor PCA, scalar-on-tensor regression, tensor-on-tensor regression, and beyond. We exploit the intrinsic geometry of rank-one tensors by casting the recovery task as an optimization problem over the Segre manifold, the smooth Riemannian manifold of rank-one tensors. This geometric viewpoint yields two powerful algorithms: Riemannian Gradient Descent (RGD) and Riemannian Gauss-Newton (RGN), each of which preserves feasibility at every iteration. Under mild noise assumptions, we prove that RGD converges at a local linear rate, while RGN exhibits an initial local quadratic phase followed by linear refinement to the statistical error. Extensive synthetic experiments validate these convergence guarantees and demonstrate the practical effectiveness of our methods.

stat.ML · cs.LG · math.OC · math.ST · stat.ME

版本更新 · v2arXiv:2505.20536

Covariate-Adjusted Deep Causal Learning for Heterogeneous Panel Data Models

Guanhao Zhou, Yuefeng Han, Xiufan Yu

原文摘要

This paper studies the task of estimating heterogeneous treatment effects in causal panel data models with covariate effects. We propose a novel Covariate-Adjusted Deep Causal Learning (CoDEAL) framework that cohesively deals with the underlying heterogeneity and nonlinearity of both panel units and covariate effects. CoDEAL integrates nonlinear covariate effect components (parameterized by a feed-forward neural network) with nonlinear factor structures (modeled by a multi-output autoencoder) to form a heterogeneous causal panel model. The nonlinear covariate component flexibly captures complex covariate influences on outcomes, and the nonlinear factor decomposition enables CoDEAL to effectively capture both cross-sectional and temporal dependencies inherent in the data panel. This latent structural information is subsequently integrated into a customized matrix completion algorithm, thereby facilitating more accurate counterfactual imputation. Moreover, the use of a multi-output autoencoder explicitly accounts for heterogeneity across units and enhances the model interpretability of the latent factors. We establish theoretical guarantees on the convergence of the estimated counterfactuals and demonstrate the compelling performance of the proposed method using extensive simulation studies and real-world data applications.

stat.ML · cs.LG · econ.EM · stat.ME

版本更新 · v2arXiv:2605.05755

Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement

Haodong Liang, Lifeng Lai

原文摘要

We investigate the ability of transformers to perform in-context reinforcement learning (ICRL), where a model must infer and execute learning algorithms from trajectory data without parameter updates. We show that a linear self-attention transformer block can provably implement policy-improvement methods, including semi-gradient SARSA and actor-critic, via explicit parameter constructions. Beyond existence, we design a teacher-mimicking training procedure, analyze its gradient-flow dynamics, and establish the first convergence guarantee in the ICRL literature: under suitable richness conditions on the training MDP distribution, gradient flow converges locally and exponentially to an optimal parameter manifold corresponding to the desired RL update. Empirically, training transformers on randomly generated tabular MDPs confirms these predictions: the learned models recover the parameter structure of our explicit constructions and, when deployed on unseen MDPs, deliver strong in-context control performance. Together, these results illuminate how transformer architectures internalize and execute classical reinforcement learning algorithms in context, bridging mechanistic understanding and training dynamics in ICRL.

stat.ML · cs.AI · cs.LG

新论文 · v1arXiv:2611.10996Apple ML Research

Toward a theory of Evaluability

Kunal Talwar

原文摘要

Machine learning models, particularly large foundation models, are increasingly evaluated along multiple axes that are relevant for their usage. In addition to accuracy measures appropriate for the specific task of interest, measures capturing safety, calibration and fairness properties are often of interest. The classical approach to these problems assumes that the test set has been held out from the training process, an assumption that is increasingly harder to justify for large foundation models. In this work, we formalize a notion of evaluability without this assumption, and ask if it is feasible to evaluate black-box models in settings where our test samples may have been used during training. We study this problem under different settings, capturing different levels of access to the model. We first study the natural setting, where the evaluator only looks at the output of the model on the test set. We fully characterize the class of distributions where accurate evaluation is possible, and show tight upper and lower bounds on the sample complexity of evaluation. Our results show that black-box evaluation in this setup is feasible if and only if the distribution is close to being small support. We then consider evaluation algorithms that can query the model on additional inputs and demonstrate connections to self-correctors studied in program testing. Using these, we show that a small amount of query access can increase the power of the evaluator for natural distributions and concept classes.

cs.LG · stat.ML

版本更新 · v2arXiv:2605.05973

Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking

Jiefu Zhang, Yang Xu, Haixiang Sun, Zihan Zhou, Tianyu Cao, Vaneet Aggarwal

原文摘要

Adaptive prompt and program search makes LLM evaluation selection-sensitive. Once benchmark items are reused inside tuning, the observed winner's score need not estimate the fresh-data performance of the full tune-then-deploy procedure. We study inference for this procedure-level target under explicit tuning budgets. We propose SIREN, a selection-aware repeated-split reporting protocol that freezes the post-search shortlist, separates splitwise selection from held-out evaluation, and uses an item-level Gaussian multiplier bootstrap for uncertainty quantification. In a fixed-shortlist regime with smooth stabilized selection, the estimator admits a first-order item-level representation, and the bootstrap yields valid simultaneous inference on a finite budget grid. This supports confidence intervals for procedure-performance curves and pre-specified equal-budget and cross-budget comparisons. Controlled simulations and MMLU-Pro tuning experiments show that winner-based reporting can be optimistic and can change deployment conclusions, while SIREN remains close to the finite-sample reporting target.

stat.ML · cs.AI · cs.LG · stat.AP

版本更新 · v2arXiv:2607.09645

Deep Gaussian Processes on Directed Acyclic Graphs

Federico L. Perlino, Oliver Hamelijnck, Adam M. Johansen, Theodoros Damoulas

原文摘要

Many real-world processes can be represented as compositions of functions along a directed acyclic graph (DAG). In causal modelling, these correspond to the underlying mechanisms; in engineering, to multiple fidelity levels; and in gene-regulatory networks, to transcription factors. These functions are partially observed across the DAG, with noisy and heterogeneously sampled measurements, posing significant challenges for reconstruction, uncertainty propagation, and inference. To tackle these challenges, we place priors over functions and naturally arrive at Deep Gaussian Processes over DAGs. We theoretically study their prior-collapse behaviour and the effect of graph topology and intermediate observations on the preservation of information. We obtain almost-sure lower bounds on the asymptotic frequency of depths at which the distinction between inputs is preserved, identify broad kernel classes for which these hold, and prove an observation by \citep{dunlop2018} on the role of input connections. We offer a structured variational approximation that retains graph dependencies, preserves compositional uncertainty, and captures the explaining-away behaviour of colliders. Finally, we empirically validate our theory and methodology at scale and under DAG misspecification. We then model a latent-collider DAG, a protein signalling network, and a multi-fidelity \mbox{heavy-ion} collision emulation task, attaining state-of-the-art performance while recovering low-fidelity contributions and yielding interpretability over the simulator hierarchy.

stat.ML · cs.LG · math.ST · stat.CO · stat.ME

版本更新 · v3arXiv:2602.00474

Persistent-Transient Policy Evaluation for Markov Chains via Minimal Peripheral Quotients

Yang Xu, Vaneet Aggarwal

原文摘要

We study fixed-policy evaluation for finite Markov chains that may be reducible and periodic. Classical evaluation methods with gain and bias decomposition are not always diagnostic: the gain records only invariant Cesàro averages, while persistent phase-dependent behavior is absorbed into the bias together with genuinely transient effects. We identify the real peripheral invariant subspace $\mathcal{K}(P)$ of the transition matrix $P$ as the source of this ambiguity. Quotienting by $\mathcal{K}(P)$ is the minimal exact quotient that removes all non-decaying modes and makes the remaining dynamics strictly stable. After choosing a gauge projection $Π$ with kernel $\mathcal{K}(P)$, the reward admits a unique decomposition $r = g_Π^\star + (I-P)v_Π^\star$, where $g_Π^\star$ is a persistent regime profile and $v_Π^\star$ is a gauge-fixed transient component. An exact comparison with classical normalized gain and bias shows that the new pair reallocates the same information into a persistent profile and a gauge-fixed representative of the transient quotient. This decomposition reconstructs finite-horizon returns, recovers statewise average reward, admits a transient-cost interpretation, and yields a stable estimator under a generative model.

stat.ML · cs.LG · math.NA

新论文 · v1arXiv:2611.10972

JUBILEE: A Benchmark Environment for Off-Policy Evaluation in Debt Collections

Diego Silva Prado, Tiago Amaral Penteado de Jesus, Bruno Caldas de Souza, Cristiano Bevitori Maffia De Oliveira

原文摘要

Off-policy evaluation (OPE) estimates the value a candidate decision policy would achieve, using only data logged by the policy already in production. In debt collections, unrecorded factors can influence both actions and outcomes. This paper presents JUBILEE, a benchmark environment built around a debt-collections simulator that generates million-loan panels with competing risks, rare recovery events, and hidden factors driving both outcomes and assignment. The evaluation includes an 80-run comparison of tabular and gradient-boosted estimators under two synthetic outcome mechanisms and a 190-run parameter grid. Independent policy simulations provide reference values with estimated Monte Carlo uncertainty for a synthetic reward score and for simulated cash. With hidden factors influencing action assignment, several tabular methods agree closely while underestimating the field-visit policy's score, an error that also appears with boosted models. Intervals from tabular doubly robust estimation usually contain the reference without hidden assignment dependence but mostly miss it when that dependence is present. Estimates from randomized decisions do not show the same large error, although their limited precision leaves policy choice uncertain. The decision consequences depend on how policy value is defined. The field-visit policy ranks last under the score and first under simulated cash, which excludes channel costs. Under cash, log-based estimators miss this best candidate in nearly every run with hidden assignment dependence in the paired study. JUBILEE supports reproducible study of these failure modes under specified collections mechanisms.

cs.LG · q-fin.RM · stat.ML

新论文 · v1arXiv:2611.10965

Bayesian transfer for offline Q-learning with multi-task Gaussian processes

Wenxing Guo

原文摘要

Offline reinforcement learning can benefit from data collected in related tasks, yet indiscriminate knowledge transfer may lead to negative transfer under task mismatch and limited target-data coverage. We propose a target-aware Bayesian transfer framework for offline Q-learning based on multi-task Gaussian processes that jointly model task-specific Q-functions. Source tasks are assessed using target predictive evidence and maximum mean discrepancy (MMD), while posterior uncertainty and behavior-supported action sets guide conservative Bellman updates and policy extraction. To account for the dependence induced by Bellman bootstrapping, we distinguish Bellman innovations from bootstrap errors and establish stability and predictive source-selection results under explicit regularity conditions. We further derive selection-adaptive posterior contraction and offline policy-performance bounds, identifying the structural conditions under which source information can be pooled to achieve minimax-optimal rates in a shared-function Gaussian sequence experiment. Simulation studies with heterogeneous source tasks demonstrate improved target Q-function estimation and reduced material negative transfer compared with indiscriminate source pooling. An application to human decision-making data further shows higher average expected immediate payoff than both target-only learning and matched-size random source selection. These results highlight the benefits of combining target-aware source selection with uncertainty-aware conservative learning for reliable and sample-efficient offline reinforcement learning.

stat.ML · cs.LG

版本更新 · v2arXiv:2605.09654

Metropolis-Adjusted Diffusion Models

Kevin H. Lam, Tyler Farghly, Christopher Williams, Jun Yang, Yee Whye Teh, Arnaud Doucet

原文摘要

Sampling from score-based diffusion models incurs bias due to both time discretisation and the approximation of the score function. A common strategy for reducing this bias is to apply corrector steps based on the unadjusted Langevin algorithm (ULA) at each noise level within a predictor-corrector framework. However, ULA is itself a \emph{biased} sampler, as it discretises a continuous diffusion process. In this work, we consider \emph{adjusted} Langevin correctors that employ Metropolis--Hastings (MH) or Barker's accept-reject steps to correct for this bias. Since the target density ratio typically required by MH-based algorithms is unavailable, we propose methods that instead utilise the score function to compute the correct acceptance probability. We introduce the first exact method for adjusting Langevin corrections in diffusion models, based on a two-coin Bernoulli factory algorithm. We also propose an efficient approximation based on Simpson's rule that achieves accuracy of order $5/2$ in the step size at near-zero marginal cost. We demonstrate that these procedures improve sample quality on both synthetic and image datasets, yielding consistent gains in Fréchet Inception Distance (FID) on the latter.

stat.ML · cs.LG · stat.CO

版本更新 · v2arXiv:2605.01136

Spectral Graph Sparsification Preserves Representation Geometry in Graph Neural Networks

Sanjukta Krishnagopal

原文摘要

Spectral graph sparsification is a classical tool for reducing graph complexity while preserving Laplacian quadratic forms. In graph neural networks (GNNs), sparsification is often used to accelerate computation while maintaining predictive performance. In this work, we study a complementary representation-level question: does sparsification preserve the geometry of learned embeddings? For polynomial-filter GNNs, we prove that any $ε$-spectral sparsifier induces $O(ε)$ perturbations in polynomial graph filters, multilayer hidden representations, and their Gram matrices. These guarantees imply stability of squared pairwise distances, class means, and covariance structure in embedding space. We further establish finite-time training stability: under smoothness and boundedness assumptions, gradient descent on dense and sparsified graphs produces weight trajectories whose separation grows at most proportionally to the sparsification distortion. Empirically, effective-resistance sparsification validates the predicted perturbation chain on synthetic graphs and preserves hidden representation geometry on real datasets. In our experiments, the gram matrix and training dynamics show low divergence even under substantial sparsification, consistent with the predicted stability under spectral sparsification. Hidden Gram preservation strongly predicts neighborhood preservation and class-centroid stability across FashionMNIST, Cora, and Paul15. Together, these results show that spectral sparsification preserves not only graph operators, but also the representation geometry that supports downstream use of GNN embeddings for interpretability.

cs.LG · cs.SI · math.SP · stat.ML

新论文 · v1arXiv:2611.10895

Robust Satisficing Ensemble: Scalable Model Aggregation Under Distribution Shifts

Ahmet Faruk Cetinkaya, Enes Agirman, Cem Tekin

原文摘要

Standard ensemble methods can improve robustness by mitigating individual model stochasticity, but they frequently fail to generalize beyond the training distribution, leaving systems vulnerable to out-of-distribution (OOD) data. We introduce the Robust Satisficing Ensemble (RSE), a scalable probabilistic aggregation framework designed to secure model combinations by minimizing their fragility against distributional changes. By bridging a robust satisficing objective with a quadratic risk surrogate, RSE derives an analytical inner adversary. This fundamental reformulation enables highly efficient weight updates without the computational bottleneck of standard minimax optimization, crucially decoupling optimization complexity from sample size to scale effectively to large datasets. In particular, we prove a geometrically decaying truncation error for the fragility approximation via Lagrange-Burmann expansion and establish stationarity guarantees for RSE. We evaluate RSE across a comprehensive spectrum of distribution shifts, including adversarial label shift (SST-5, TREC), natural subpopulation and temporal shifts (CivilComments, HuffPost). Our results demonstrate that RSE consistently outperforms strong robust baselines, including GroupDRO, IRM, and SRM. These findings establish RSE as a practical, theoretically grounded safeguard for reliable model aggregation in dynamic environments.

cs.LG · cs.AI · stat.ML

版本更新 · v2arXiv:2605.31504

When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework

Dylan Steiner, Gustavo Arango-Argoty, Gerald Sun, Etai Jacob

原文摘要

Multimodal models in oncology can produce accurate predictions, but accurate prediction does not reveal whether the model has learned biology that is shared across modalities, biology confined to one modality, or spurious correlations that reflect confounders rather than genuine biology. We introduce DECAT, a model-agnostic post-hoc evaluation framework that classifies multimodal representations into four diagnostic scenarios (shared, spurious, modality-specific, or no signal) for a given task and modality, using five null-referenced metrics and a rule-based decision procedure. The framework operates on learned representations, requires no knowledge of which specific confounder is present, and returns indeterminate when the evidence is insufficient. We evaluate multimodal architecture properties using DECAT on synthetic data across four multimodal model classes spanning over 2,500 trained representations, and demonstrate its operational use on 8,979 TCGA patients with both multimodal model embeddings and five pretrained pathology foundation models. Entangled models (e.g., CLIP) detect shared signal near-perfectly on synthetic data but also falsely claim it when it is absent, doing so in the majority of such cases on real foundation model embeddings. This false claim rate increases with signal strength and sample size, so that larger cohorts and stronger representations produce more confident but still incorrect diagnoses. Applied to both multimodal TCGA embeddings and five pathology foundation models without paired RNA, DECAT detects confounding invisible to AUROC without requiring the confounder labels, as confirmed by post-hoc stratification.

cs.LG · stat.ML

版本更新 · v4arXiv:2506.15643

Understanding Randomization in Greedy Model Search

Xin Chen, Jason M. Klusowski, Yan Shuo Tan, Chang Yu

原文摘要

We study feature subsampling in greedy model search, a randomization mechanism motivated by random forests and other ensemble methods. Although recent theory draws an analogy between this randomization and ridge regularization, these results largely rely on base learners optimized via ordinary least squares (OLS). We therefore investigate the effects of feature subsampling on greedy forward selection, a tractable abstraction of the adaptive split search used by decision trees. Assuming an orthogonal design, we prove that, under suitable conditions, ensembling with a fixed feature subsampling size can reduce expected training error without increasing degrees of freedom, thereby reducing fixed design prediction error. We also show that both the training error and degrees of freedom need not be monotone in the subsampling rate, breaking the analogy with standard shrinkage methods like the lasso or ridge regression under orthogonality. Furthermore, we characterize the exact asymptotic behavior of the estimator, showing that it adaptively reweights OLS coefficients based on their rank, with weights that are bounded by explicit logistic functions.

stat.ML · cs.LG

版本更新 · v2arXiv:2608.26552

Hadamard Flattening and Gaussian Pooling Sketch for Least Squares with Coordinate-wise Guarantee

Zhao Song, Lichen Zhang

原文摘要

Randomized sketch-and-solve algorithms accelerate overconstrained $\ell_2$ regression by replacing the input with a much smaller, randomly projected problem. While standard subspace embeddings nearly preserve the regression cost, coordinate-wise accuracy is more delicate: rather than the objective value, we want the solution vector itself to be close to the optimum in $\ell_\infty$ norm. Specifically, we seek $x'\in\mathbb{R}^d$ with $\|x'-x^\star\|_\infty\leq\fracε{\sqrt d}\|Ax^\star-b\|_2\|A^\dagger\|_{\mathrm{op}}$, where $A\in\mathbb{R}^{n\times d}$ is the design matrix, $b\in\mathbb{R}^n$ the label vector, $x^\star$ the optimal solution and $A^\dagger$ the pseudo-inverse of $A$. Price, Song and Woodruff introduced this problem and claimed that the subsampled randomized Hadamard transform (SRHT) with $O(ε^{-2}d^{1+Θ(\sqrt{\log\log n/\log d})})$ rows achieves this guarantee; we point out a gap in the proof. Song, Ye, Yin and Zhang later claimed $O(ε^{-2}d\log^3 n)$ rows, but their proof relies on an independence assumption that does not hold in general, and we exhibit an explicit instance on which it fails. To achieve a truly nearly-linear-in-$d$ row count, we introduce a new fast, dense randomized transform combining a randomized Hadamard flattening, a random permutation, and balanced, disjoint Gaussian pooling. Conditioned on the Hadamard-and-permutation stage, the sketched problem becomes an exact Gaussian regression whose noise is independent of the entire sketched design, exactly what Song, Ye, Yin and Zhang's argument was missing. Our sketch yields the $\ell_\infty$ guarantee with $m=O(ε^{-2}d\log d)$ rows, using one Hadamard pass with padded internal dimension $N=\widetilde{O}(n+ε^{-2}d^3)$. It is also efficient to apply: $(SA,Sb)$ can be computed in $O(Nd\log N)=\widetilde{O}(nd+ε^{-2}d^4)$ time.

cs.DS · cs.LG · stat.ML

版本更新 · v3arXiv:2604.19530

Calibrating Scientific Foundation Models with Inference-Time Stochastic Attention

Akash Yadav, Taiwo A. Adebiyi, Ruda Zhang

原文摘要

Transformer-based scientific foundation models are increasingly deployed in high-stakes settings, but current architectures give deterministic outputs and provide limited support for calibrated predictive uncertainty. We propose Stochastic Attention, a lightweight sample-average inference-time modification that randomizes attention by replacing softmax weights with normalized multinomial samples controlled by a single concentration parameter, and produces predictive ensembles without retraining. To set this parameter, we introduce a calibration objective that matches the stochastic attention output with the target, yielding an efficient univariate post-hoc tuning problem. We evaluate this mechanism on scientific foundation models for weather and time-series forecasting, as well as a regression task. Across benchmarks against baseline methods, Stochastic Attention achieves the strongest native calibration and the sharpest prediction intervals at comparable calibration, with adaptation costs nearly three orders of magnitude lower than the next-best baseline. Code is available at https://github.com/UQUH/StochasticAttention.

cs.LG · cs.CE · stat.ML

版本更新 · v3arXiv:2410.23244

Very fast Bayesian Additive Regression Trees on GPU

Giacomo Petrillo

原文摘要

Bayesian Additive Regression Trees (BART) is a nonparametric Bayesian regression technique based on an ensemble of decision trees. It is part of the toolbox of many statisticians. The overall statistical quality of the regression is typically higher than other generic alternatives, and it requires less manual tuning, making it a good default choice. However, it is a niche method compared to natural competitors such as XGBoost, due to the longer running time, making sample sizes above 10000-100000 a nuisance. We present a GPU implementation of BART, up to 500x faster than the previous state of the art, making BART fast enough to be a viable option on large datasets, and showing that BART parallelizes better than its non-Bayesian counterparts. We also find that, to be as practical as tree boosting on large datasets, BART would need a scalable way to infer the number of trees, which we do not provide. Our implementation is available in the Python package bartz.

stat.ML · cs.LG

新论文 · v1arXiv:2611.10803

On Generalization in Bilevel Least Squares with Overparameterized Neural Networks

Fares El Khoury, Edouard Pauwels, Samuel Vaiter, Michael Arbel

原文摘要

Bilevel optimization addresses machine learning problems in which the outer objective is a value function implicitly defined through the solution of an inner problem. While its algorithmic aspects are well developed, its generalization properties remain much less understood. In this paper, we study the generalization of inexact gradient methods when both objectives are least-squares regression problems and when the estimator of the value function's gradient, or hypergradient, is obtained by implicit differentiation in function space. This approach allows us to derive stationarity bounds for these methods in terms of the uniform generalization errors of the two regression functions needed to estimate the hypergradient. When these regression functions are defined as overparameterized two-layer ReLU networks trained by gradient flow with early stopping, we obtain explicit finite-sample bounds on these generalization errors in the Neural Tangent Kernel (NTK) regime. Combining these with our general inexact gradient analysis shows the effect of sample size on the overall convergence of the bilevel method. Numerical experiments on synthetic data illustrate our results. Beyond the NTK regime, further experiments reveal an initialization-dependent implicit bias induced by the bilevel structure, which can lead to feature learning at the inner level and overall performance improvements.

stat.ML · cs.LG

新论文 · v1arXiv:2611.10787

Trustworthy Recursive Self-Improvement with Statistical Guarantees

Zihe Gao, Zhengling Qi, Yifan Cui

原文摘要

Recursive self-improvement offers a promising path toward AI agents that improve through experience by refining their own harnesses without retraining their underlying large language models (LLMs). Making this self-improvement process reliable, however, requires more than generating new updates: agents must decide which changes are worth keeping. A greedy gate that accepts updates based solely on higher validation scores can mistake an observed gain for genuine progress and lead to performance degeneration, whereas overly conservative statistical gates may reject beneficial updates and hinder performance improvement. We introduce Trustworthy Recursive Updates with Statistical Testing (TRUST), a framework that adapts an online multiple hypothesis testing procedure to control the rate of false evolution across rounds, with an LLM analyzing historical context to help determine how much evidence is required to update the incumbent. Under interpretable assumptions, we prove that TRUST limits the proportion of accepted updates that result in unacceptable performance loss. We further derive a lower bound on the final performance gain. Across the benchmarks HotpotQA, DROP, and APPS with the models DeepSeek-V4-Flash and Qwen-3.8-Flash, TRUST reduces the average ratio of accepted inferior updates from the greedy gate's 45.59% to 2.78%, while achieving higher final scores than the state-of-the-art statistical gate baselines for recursive self-improvement. These findings demonstrate that TRUST strikes a balance between statistical reliability and performance gain in recursive self-improvement.

stat.ML · cs.AI · cs.LG · stat.ME

版本更新 · v3arXiv:2606.14095

Lyapunov-Based Sample Complexity Analysis for Weakly-Coupled MDPs

Tianhao Wu, Matthew Zurek, Weina Wang, Qiaomin Xie

原文摘要

We study the sample complexity of learning in average-reward weakly-coupled Markov decision processes (WCMDPs) and Restless Bandits (RBs) under a generative model. Naive reduction to a tabular MDP leads to high complexity bounds as the state-action space is exponentially large in the number of arms $N$. By exploiting the weakly coupled structure, we show that near-optimal policies can be learned with sample and computational complexities that are polynomial in $N$. Specifically, we analyze the plug-in approach, which applies an efficient planning algorithm to an empirical model estimated from data. For fully heterogeneous WCMDPs, we establish the first finite-sample PAC guarantee with polynomial complexity and an $O(1/\sqrt{N})$ optimality gap. For homogeneous RBs, we further prove that a smaller optimality gap is achievable under mild structural assumptions. A primary technical contribution is a novel Lyapunov-based analysis framework. Unlike classical approaches that rely on the difficult-to-control bias function, our framework uses an explicitly constructed Lyapunov function along with a drift transfer technique between the true and empirical models. A key step of independent interest in our framework is a fine-grained perturbation analysis for the underlying linear programming (LP) relaxation, which provides a general tool for analyzing LP-based policies and weakly-coupled systems.

cs.LG · math.OC · math.PR · stat.ML

新论文 · v1arXiv:2611.10738

Matchability-Aware Conformal Prediction for Open-Ended Language Model Generation

Ilia Azizi

原文摘要

Large language models (LLMs) often do not produce calibrated uncertainty estimates for their answers. We propose CAELUM, a framework that builds semantic prediction sets while separating aleatoric from epistemic uncertainty. CAELUM samples answers using an ensemble of clarification prompts, clusters them by semantic equivalence, and estimates the aleatoric and epistemic components via kernel-based von Neumann entropy before calibrating nonconformity scores with split conformal prediction. We use a weight $w_λ(x)=1+U_{\mathrm{ale}}(x)+λ\,U_{\mathrm{epi}}(x)$ to control for the contribution of the epistemic uncertainty of each instance and fit a coefficient $λ^\star$ to minimize prediction set size. When the objective is nonflat and selection is stable across splits, $λ^\star$ could be a diagnostic, where large values may indicate epistemic dominance (retrieve, abstain) and $λ^\star\to 0$ aleatoric dominance (clarify). For individual queries, a log ratio $R(x)$ compares the weighted epistemic component with the aleatoric component. Theoretically, we demonstrate that the coverage is capped by matchability, the fraction of items whose reference answer appears in any sampled cluster. Empirically, this matchability can be predicted from the original question features using a lightweight classifier. We use those predictions to define a stratum and derive a conformal coverage bound within it. Across 30 LLM and dataset combinations (4-32B), mean coverage is 0.74 in the learned stratum versus 0.65 overall, averaged over the 78 evaluations using the learned classifier. Project page: https://unco3892.github.io/caelum/

stat.ML · cs.CL · cs.LG · stat.ME

版本更新 · v2arXiv:2610.32740

Strongly Adaptive Online Learning with Time-Varying Movement Costs

Andrew Jacobsen, Emmanuel Esposito, Hao Qiu, Mengxiao Zhang

原文摘要

We study online convex optimization (OCO) with time-varying movement costs, where the learner incurs both a loss and a time-dependent penalty for changing its decisions. While previous work has established dynamic regret guarantees for this setting, obtaining strongly adaptive guarantees that hold simultaneously over arbitrary intervals remains an open challenge. We address this by developing a new algorithm which, for every interval $[a,b]\subseteq[T]$ and arbitrary comparator sequence, achieves a data-dependent regret bound of $ \widetilde{\mathcal{O}}\left( \sqrt{(D^2+DP_{[a,b]})\sum_{t \in [a,b]}(\|g_t\|^2+λ_t\|g_t\|)} \right) $, where $D$ is the diameter of the decision domain, $P_{[a,b]}$ is the path length of the comparator sequence over $[a,b]$, $g_t$ is the gradient of the loss, and $λ_t$ is the movement-cost coefficient at round~$t$. Key to our approach is a novel reduction to sleeping experts with movement costs, a previously unexplored variant that is of independent interest. As a result of our development, we further obtain strongly-adaptive dynamic regret guarantees under arbitrary delayed feedback with a refined dependence on interval-restricted cumulative delays. Our approach also naturally extends to OCO with time-varying memory, where we obtain strongly-adaptive guarantees which improve the best-known bounds.

cs.LG · stat.ML

版本更新 · v2arXiv:2605.10163

Coarsening Linear Non-Gaussian Causal Models with Cycles

Francisco Madaleno, Francisco C. Pereira, Alex Markham

原文摘要

Recent work on causal abstraction, in particular graphical approaches focusing on causal structure between clusters of variables, aims to summarize a high-dimensional causal structure in terms of a low-dimensional one. Existing methods for learning such summaries from data assume that both structures are acyclic. Acyclicity is helpful for causal effect identification and reasoning, but it excludes the many systems of scientific interest that contain feedback loops. We show that in the linear non-Gaussian (LiNG) setting, the acyclicity assumption on the high-dimensional structure can be relaxed while still allowing recovery of a low-dimensional causal directed acyclic graph (DAG). LiNG models with cycles are observationally identifiable only up to an equivalence class whose members differ only within their feedback loops. Our low-dimensional DAG, which is invariant across all members of this class, is therefore identifiable from observational data alone. While existing approaches for learning this equivalence class have exponential time complexity, our low-dimensional summary is learned in worst-case cubic time and comes with finite-sample guarantees. We provide open source code and experiments on synthetic data to corroborate our theoretical results.

stat.ML · cs.AI · cs.LG

新论文 · v1arXiv:2611.10691

A Geometric Theory for Nonequilibrium Transformer Dynamics

Aythami Bethencourt de León, Darryl D. Holm

原文摘要

We develop a variational and geometric theory based on statistical physics for the joint evolution of Transformer attention and token representations at fixed model parameters. To this end, we introduce a free energy for the attention and model the evolution of the attention weights by its Fisher--Rao gradient flow. At fixed token representations, we prove that the softmax matrix is the unique free-energy minimiser and that every positive attention trajectory converges to it. For prescribed time-dependent representations, we show that the attention dynamics becomes linear after a change to logit variables. This result yields tracking bounds relative to the instantaneous softmax and sufficient conditions for attention to converge to a fixed matrix. Hamiltonian dynamics for the representations and their momenta are then coupled to the attention relaxation driven by the same energy. This coupling results in an energy-dissipation law. We also develop two alternative Hamiltonians incorporating the attention value map. We prove global well-posedness of the attention flow and all three coupled systems.

cs.LG · math-ph · math.PR · nlin.AO · stat.ML

版本更新 · v3arXiv:2512.23818

Energy-Tweedie: Score meets Score, Energy meets Energy

Andrej Leban

原文摘要

Denoising and score estimation are classically linked through Tweedie's formula, which relates the posterior mean under Gaussian noise to the Stein score of the noisy marginal. In this work, we extend this perspective beyond Gaussian noise to a broad class of Gibbs (energy-based) noise distributions, with the generalized Gaussian family as an example. We derive the Energy-Tweedie identity: when the denoising posterior is viewed through the lens of scoring rules, the path derivative of a kernel scoring rule defined by the noise potential recovers the Stein score of the noisy marginal. Thus, the familiar correspondence between Gaussian noise, posterior means, squared loss, and Tweedie's formula is lifted to a distributional correspondence between Gibbs noise distributions, full posterior laws, kernel scoring rules, and the Energy-Tweedie identity, which gives a Tweedie-style relation for each noise potential. Among its consequences, this identity gives a posterior-samples-to-score route to score estimation, yields a principled criterion for estimating unknown noise parameters, and enables diffusion-style sampling along user-chosen paths through the noise-parameter space, supplying the score-based perspective on recent generative methods trained with scoring rules.

stat.ML · cs.LG

版本更新 · v2arXiv:2606.31963

Signed-Permutation Coordinate Transport for RMSNorm Transformers

John Sweeney

原文摘要

Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, LoRA updates, optimizer states, and merge alignments. This requires fixing the model's residual-stream gauge. We show that the native coordinate-preserving orthogonal gauge is architecture-dependent: LayerNorm residual charts have permutation gauge $S_d$ up to a global sign flip, while RMSNorm charts with generic gains have signed-permutation gauge $B_d = S_d \ltimes \{\pm 1\}^d$. Permutation-only alignment is thus symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove that with decorrelated coordinates, signed-correlation matching has a permutation-accuracy ceiling equal to the true gauge's positive-sign fraction up to $O(d 2^{-d})$, which sign-marginalized matching removes. Omitting signs induces expected squared error $2\|ΔW\|_F^2$ on weight updates and loses orientation on vectors and AdamW first moments, while RMSNorm gains and AdamW second moments permute without signs. Controlled tests confirm these laws: a Qwen2.5-1.5B SST-2 LoRA adapter retains 0.944 accuracy (from 0.945) under $B_d$ transport vs. 0.523 under $S_d$, and steering preserves effects to $<10^{-4}$ at zero raw top-10 overlap. Across natural checkpoints, inherited slots suffice for same-base descendants (96.2%-100% retention), whereas cross-generation pairs support sparse prompt-stable subsets with causally decisive negative signs ($+0.185$ vs. $-0.010$ at $8σ$ on Qwen2/2.5-0.5B; $+0.189$ vs. $-0.021$ on 1.5B). Along fine-tuning trajectories, composing local $B_d$ gauges recovers 91.1% of cross-run Qwen2.5-1.5B coordinates at 1,500 steps vs. 60.3% for endpoint matching. Merge and interpretability audits confirm that coordinate names are reproducible only relative to an explicit $B_d$ gauge.

cs.LG · cs.CL · stat.ML