跳到正文
研踪PaperTrail

论文库

检索计算机领域论文,按研究方向、作者或关键词筛选。

部分采集异常
正在读取账号与研究记录…

全站更新概况

+758新增论文
+578其中 AI 相关
+94已有论文修订
+95新增 AI 导读

arXiv 最近完成扫描:2026-11-17 04:58 · 全站最近入库:2026-11-17 04:58各信源进度 →

统计口径与数据新鲜度

统计 2026-11-16 07:47 — 2026-11-17 07:47(北京时间) · 点击数字查看对应简报;AI 相关包含在新增论文内。

最近完成检查:2026-11-17 04:58最近发现新增版本:2026-11-17 04:58库内论文最新更新时间:2026-11-14 02:59

每分钟刷新已保存的数据。空检查不代表采集异常;新收录可能包含历史论文。AI 导读按新增版本计数。

24,106 已收录论文20,158 AI 相关7,049 有修订112 HF 社区发现

浏览论文库

时间范围跟随排序:版本更新、本站收录或首次发表。最新收录可能包含历史论文。部分信源异常,查看状态 →

找到 806 篇论文 · 按版本更新时间排序上轮扫描起点 2026-11-17 04:57(北京时间)
版本更新 · v2arXiv:2605.07907

Consistency Regularised Gradient Flows for Inverse Problems

Alessio Spagnoletti, Tim Y. J. Wang, Marcelo Pereyra, O. Deniz Akyildiz

原文摘要

Vision-Language Latent Diffusion Models (LDMs) (Rombach et al., 2022) provide powerful generative priors for inverse problems. However, existing LDM-based inverse solvers typically require a large number of neural function evaluations (NFEs) and backpropagation through large pretrained components, leading to substantial computational costs and, in some cases, degraded reconstruction quality. We propose a unified Euclidean-Wasserstein-2 gradient-flow framework that jointly performs posterior sampling and prompt optimization in the latent space through a single flow that aligns the prior and posterior with the observed data. Combined with few-step latent text-to-image models, this formulation enables low-NFE inference without backpropagation through autoencoders. Experiments across several canonical imaging inverse problems show that our method achieves state-of-the-art performance with significantly reduced computational cost.

stat.ML · cs.CV · cs.LG

版本更新 · v3arXiv:2609.15825

Sharp Regret Bounds and a Task-Covariance Correction for Spectral Representation Learning

Dier Tang, Jing Yee Tan, Guangyue Han

原文摘要

Spectral contrastive learning keeps the $k$ directions along which two views of the data depend most strongly, while downstream tasks may need other directions. Existing guarantees for spectral features assume that the distribution of tasks has no preferred direction. We show that this isotropy condition is the wrong criterion. With linear probing, the task distribution affects the average probing gain only through the matrix $BΛB^\top$, where $B$ records the dependence between views and $Λ$ is the task covariance: optimal features span its top-$k$ eigenspace, whereas spectral features span that of $BB^\top$. Transfer therefore depends on alignment, and strongly uneven task preferences cost nothing when they favour the spectral directions. Otherwise the regret of spectral features is at most linear in the anisotropy of $Λ$. No representation chosen without task information improves this rate when the leading spectrum is flat, and a spectral gap makes it quadratic. With labelled tasks, the same matrix gives a correction: Fisher discriminant directions for a frozen encoder, or a one-term change to the spectral contrastive loss. We bound the regret of the correction by the error in estimating $BΛB^\top$, and compare this error with the misalignment of spectral features to decide when to correct. The analysis extends to finite samples, continuous variables and noisy views. Correction reduces synthetic held-out regret from $0.858$ to $0.003$ with $2000$ labelled tasks, and the decision rule gives the lowest regret among the tested rules at every bank size. On CIFAR-100, spectral features of frozen encoders keep most of the task signal on unseen classes, and where a bank of other classes would lower accuracy, the rule keeps them in every case.

cs.LG · cs.IT · stat.ML

新论文 · v1arXiv:2611.12268

Odds-Structured Distributional Regression for Interpretable Time-to-Event Prediction

Jakob Versnjak, Manuel Welte, Titus Kuehne, Marcus Kelm, Grégoire Montavon

原文摘要

Time-to-event prediction is fundamentally challenged by partial observability and aleatoric uncertainty. In clinical medicine, for instance, the timing of disease progression or acute readmission is contingent on highly dynamic and largely latent health trajectories. Rather than forcing a deterministic point forecast, distributional regression offers a principled means to address this complexity by predicting a probability distribution over time. However, explaining these distributional predictions presents a key obstacle: raw distribution parameters output by the model are abstract mathematical quantities that do not align well with domain concepts. To gain interpretability, we propose a novel paradigm where output parameters are designed to be additive in log-odds---a widely established measure in clinical risk modeling. Our approach, called ODDS-DR, moves beyond technical convenience to deliver clear domain alignment as well as the layer-wise computational structure required for robust, time-dependent feature attribution. Validated across two real-world clinical cohorts, our framework matches state-of-the-art predictive performance while resolving distributional outputs into interpretable risk markers across flexible prognostic horizons.

cs.LG · stat.ML

新论文 · v1arXiv:2611.12248

A method to automatically discover symbolic local learning rules

Andrea Perin, Fabio Anselmi, Stephane Deny

原文摘要

We present a general method for discovering symbolic local learning rules that can replace backpropagation, applicable, in principle, to any deep learning architecture and task. The method automatically discovers local loss functions specific to each layer by (1) constructing a dictionary of monomials of local tensor variables, inspired by Penrose diagram notation, (2) regressing over coefficients weighting these monomials to build layerwise polynomial losses whose gradients together minimize a target global loss. We test the method on classification, reconstruction, and self-supervised learning tasks with small multilayer networks and minimal datasets. Across all tasks, the discovered rules induce non-trivial interactions between layers, which lead to significant reductions in the corresponding global losses. Furthermore, the method selects sparse combinations of monomials, finding local rules that are more interpretable than those generated by MLP-based alternatives. The potential implications of our method range from energy-efficient deep learning to the discovery of biologically plausible learning mechanisms.

cs.LG · cs.NE · stat.ML

新论文 · v1arXiv:2611.12156

The balance between feature learning and collapse in dynamical systems-based generative models

Julian Brandon, Bruno Loureiro, N Alex Cayco-Gajic, Arthur Pellegrino

原文摘要

Over the past decade, a growing body of work has established that implicit biases in learning dynamics fundamentally shape the solutions found by neural networks, governing their learned features and generalisation performance. However, despite these strides in deep learning theory, far less is known about representation learning in generative models based on dynamical systems, from normalising flows to diffusion. Importantly, empirical evidence suggests that continuous-time generative models can be prone to mode collapse, yet a learning theory explaining such biases remains elusive. To address this, we use tools from dynamical systems and operator theory to express the gradient flow of a general class of stochastic differential equations as a composition of linear operators, allowing us to relate collapse to low-rank biases in generative models. In variational inference and flow-matching tasks, we show that this bias can manifest in two different mechanisms: i) where collapsed models are stable fixed points of the gradient flow, or ii) in sequential feature learning, where smaller modes take progressively longer to be learned, producing an effective collapse for a fixed training budget. Together these findings point to a trade-off between feature learning and collapse, suggesting an operator view as a step towards a unified framework for understanding the learning dynamics of dynamical systems-based generative models.

stat.ML · cs.AI · cs.LG · math.DS

版本更新 · v2arXiv:2602.12923

Annealing in variational inference mitigates mode collapse: A theoretical study on Gaussian mixtures

Luigi Fogliani, Bruno Loureiro, Marylou Gabrié

原文摘要

Mode collapse, the failure to capture one or more modes when targeting a multimodal distribution, is a central challenge in modern variational inference. In this work, we provide a mathematical analysis of annealing based strategies for mitigating mode collapse in a tractable setting: learning a Gaussian mixture, where mode collapse is known to arise. Leveraging a low dimensional summary statistics description, we precisely characterize the interplay between the initial temperature and the annealing rate, and derive a sharp formula for the probability of mode collapse. Our analysis shows that an appropriately chosen annealing scheme can robustly prevent mode collapse. Finally, we present numerical evidence that these theoretical tradeoffs qualitatively extend to neural network based models, RealNVP normalizing flows, providing guidance for designing annealing strategies mitigating mode collapse in practical variational inference pipelines.

stat.ML · cs.LG

新论文 · v1arXiv:2611.12059

A Black-Box Reduction from Regret to Multi-Level Coverage

Tuo Liu, Edgar Dobriban, Francesco Orabona

原文摘要

Online Conformal Prediction (OCP) sequentially constructs prediction sets whose average empirical coverage converges to a target level over time, for arbitrary, potentially adversarial datasets. In the multi-level setting, prediction sets are produced at $d$ coverage levels at once. Prior work argued for the use of online gradient descent with isotonic projections, which produces sets nested across levels, but with a $\sqrt{d}$ dependence in the per-level coverage convergence rate. Here, we both generalize and improve prior work. First, we show that isotonic projections can be used for the construction of multi-level online conformal prediction sets from \emph{any} online learning algorithm (not just gradient descent). Then, through a black-box reduction, we show that the linearized regret of the online algorithm itself automatically gives control over both per-level coverage and distributional consistency of multi-level predictions via Fenchel duality. Finally, by using the Weighted Interval Score, a standard scoring rule for assessing multi-level interval forecasts, we prove that these projections reduce both the size of the sets and their under-coverage. We study two instantiations of our theory, $p$-norm Online Mirror Descent and coordinate-wise Universal Portfolio methods, both improving the dependence on the number of levels to $\sqrt{\log d}$ for the multi-level OCP problem. Experiments on several datasets show favorable performance compared to baselines.

stat.ML · cs.LG

新论文 · v1arXiv:2611.12031

Scale-Invariant Empirical-Bayes Laplace Approximation for ReLU Networks

Shivam Pal, Piyush Rai

原文摘要

ReLU networks are well-known to have continuous scale symmetries; infinitely many reparameterized weight vectors that represent the same function. Dinh et al. (2017) have shown that, by reparameterizing, the geometry of the loss landscape (e.g., curvature) of these networks can be changed arbitrarily without affecting the underlying function. Thus, any quantity that depends on the network's geometry should be defined with care. In this paper, we show that the Empirical Bayes (EB) Laplace's approximation is not well-defined for ReLU networks, mainly due to Hessian dependence in computing the marginal likelihood, which is not invariant under ReLU scale symmetries. Thus, we can get different prior precisions, posterior covariances, and predictive uncertainties for a network having identical predictions under ReLU scaling. We define family of prior precision fields indexed by $γ$ and introduce $γ$-canonicalization, which first converts the network to its canonical representation by balancing the ratio of incoming to outgoing weights, and then fits the Empirical Bayes (EB) Laplace to the canonical representation. This procedure is equivalent to transporting the isotropic Gaussian prior from the canonical representation to the original coordinates, by producing a ReLU-scale-compatible precision field. This makes the marginal likelihood invariant under ReLU scaling, while allowing $γ$ to be selected by maximizing the marginal likelihood. The resulting EB Laplace approximation is well-defined on ReLU scale-equivalence classes and yields consistent prior precision selection and predictive uncertainties across functionally identical parameterizations.

cs.LG · stat.ML

新论文 · v1arXiv:2611.11973

Beyond Prediction Residuals: Conformal Regret Calibration for Robust Contextual Linear Optimization

Lingjie Zhao, Hansheng Jiang, Wei Qi

原文摘要

Predict-then-optimize systems convert contextual forecasts into decisions, yet conformal robust optimization typically calibrates prediction residuals rather than downstream loss. For linear optimization, this parameter-space geometry can be badly misaligned with decision risk: large objective perturbations may preserve the optimizer, while small boundary-crossing perturbations may cause substantial regret. We introduce Conformal Regret Sets, a post-hoc, model-agnostic framework that conformalizes an optimization-derived regret surrogate and uses its calibrated level set for minimax-regret refinement. We first show that directly conformalizing exact regret, although sufficient to certify the nominal predict-then-optimize decision, yields a degenerate robust problem: absent an exogenous bounded support, every alternative decision has unbounded worst-case regret. We therefore use SPO+, a pointwise upper bound on exact regret, as the conformal score, obtaining distribution-free finite-sample marginal regret guarantees under exchangeability. For contextual linear programs, the resulting set decomposes into polyhedral pieces aligned with the normal fan of the feasible polytope. A vertex is active if and only if its predicted suboptimality gap is at most half the calibrated threshold $η$, enabling local, output-sensitive generation of all relevant pieces. Moreover, if $Γ$ is the predicted gap to the closest competing vertex, the nominal and refined decisions admit the sharper worst-case-regret certificate $(η-2Γ)_+$, with an exact early exit when $η\leq 2Γ$. Experiments on contextual fractional knapsack validate the geometry and certificates: continuous refinement reduces the $0.95$-quantile and $\operatorname{CVaR}_{0.95}$ of regret by $9.1\%$ and $13.1\%$ relative to nominal PTO, while vertex-restricted refinement changes only $10.0\%$ of nominal decisions.

stat.ML · cs.LG

版本更新 · v3arXiv:2606.16138

Closing the Approximation Gap in Simulation-free Latent SDEs

Henry D. Smith, Brian L. Trippe, Scott W. Linderman

原文摘要

Recovering dynamical systems from noisy observations is a recurring challenge across scientific domains, including neuroscience and physics. Latent stochastic differential equations (SDEs) address this by modeling the system as an unobserved state that evolves according to a learnable SDE and generates the observations. Variational inference (VI) provides a tractable objective for fitting latent SDEs. Traditional VI algorithms evaluate this objective by numerical simulation over a time discretization, trading fidelity for computational cost. A recent class of algorithms, simulation-free VI, sidesteps this tradeoff by parameterizing the posterior through its instantaneous marginals rather than its drift. In this work, we show that the efficiency of existing simulation-free VI algorithms comes at a price: their parameterizations restrict the approximate posterior to a subset of the SDEs available to simulation-based methods, degrading posterior inference and parameter learning. We propose Helmholtz-SDE, a simulation-free VI algorithm that closes this gap by optimizing over path laws compatible with a prescribed collection of marginals. Helmholtz-SDE recovers dynamics more faithfully than prior simulation-free methods, with the largest gains under high posterior uncertainty. It further matches the performance of simulation-based VI at a fraction of the runtime. Code is available at https://github.com/lindermanlab/helmholtz-sde .

stat.ML · cs.LG

新论文 · v1arXiv:2611.11852

Identifiable but Hard to Estimate: Rotation in Equally Spaced Overcomplete ICA

Wenjun Xiao, Yinghao Fang, Hanxin Chen, Sihan Jia, Vince D Calhoun, Yuda BI

原文摘要

How many observations are needed to estimate a common rotation in two-dimensional overcomplete independent component analysis? We study equally spaced mixing columns with known source count, common source law, and column norm. For a fixed non-Gaussian fourth-order Gram-Charlier source law, the rotation is identifiable modulo the column spacing. Yet the sample size for fixed relative angular accuracy grows exponentially with the source count. We characterize this sample size, up to absolute constants, by the reciprocal of the squared Hellinger distance between two rotations separated by half the column spacing. An explicit radially weighted phase estimator attains this scale uniformly over the rotation family, using the observation radius to prevent cancellation in the mean of an angular harmonic. We also determine the exact exponential decay rate of single-observation total variation for the same pair. This gives bounds on the sample size exponent; its exact value remains open. Finally, a noisy binary-source example shows how an exact zero line of the observed characteristic function can have an exponentially small normal derivative.

cs.LG · cs.IT · stat.ML

新论文 · v1arXiv:2611.11847

Severity-Controlled Prediction Sets for Medication Recommendation

Yu Gu, Zijun Yu, Chi-Kuang Yeh, Xinyu Wang, Ziyang Song

原文摘要

Medication recommendation is a multi-label prediction task. However, existing prediction-set evaluation and calibration methods rely mainly on exact-overlap metrics and therefore treat all missed medications as equally severe, overlooking how closely the prediction set matches them within the medication hierarchy. To address this, we introduce Conformal Severity Control (CSC), a post-hoc framework for calibrating medication prediction sets under hierarchy-aware mismatch severity. CSC uses the Anatomical Therapeutic Chemical (ATC) hierarchy to quantify mismatch severity and controls patient-level severity without retraining the predictor, supporting both average severity and upper-quantile severity for the highest-severity mismatches. Experiments on MIMIC-IV show that CSC tracks user-specified severity tolerances and shifts prediction sets away from distant mismatches toward exact or close matches.

cs.LG · stat.AP · stat.ML

新论文 · v1arXiv:2611.11826

How Path Integral Diffusion Decides a Finite Spin Glass

Michael Chertkov

原文摘要

When an AI diffusion-type generator samples a frustrated spin glass, which macroscopic alternatives does it decide first, and when? We study this for bridge-diffusion models of the Path Integral Diffusion (PID) type started from a point source. We show that every isotropic PID is a time-changed Gaussian stochastic localization, and conversely every signal-to-noise schedule is realized by a PID protocol. What a continuous-time generator decides therefore does not depend on the protocol, which only sets when and at what cost. We describe the generator's state by its partial commitments: the probabilities it assigns, at each moment, to nested classes of configurations fixed by the target. The partial commitments are martingales that move only along class contrasts. Their ensemble average gives the information about each level at an exact rate set by squared magnetization contrasts. Each split is decided, approximately, like a binary Gaussian channel at signal-to-noise ratio 1.04/λ_P, with λ_P the leading eigenvalue of the parent's correlation matrix. The theory is applied to Bethe-lattice and Sherrington--Kirkpatrick (SK) glasses with up to 64 spins. It is observed generically that a trajectory first shifts its weight to one of the two mirror halves of the zero-field measure and then to the glassy classes inside it. These classes appear near the glass transition and keep their identity on cooling. Nested glassy levels have similar λ_P and are decided together, so a trajectory commits in one or two distinct events---the mirror choice and a compound glassy decision---which separate further with coupling and size, as the spectra predict. For a generator that takes finitely many steps, the protocol matters through the late, local stage rather than the macroscopic decisions.

cond-mat.dis-nn · cond-mat.stat-mech · cs.AI · stat.ML

新论文 · v1arXiv:2611.11819

When Should AI Overturn a Human Decision? Controversy at the 2026 World Cup and a $\sqrt{2/π}$ Solution

Amir Ali Ahmadi, Parsa Alavi, Radeen Alavi

原文摘要

We study how to combine a human binary call (a referee's offside or onside decision) with an AI numerical measurement (VAR's estimate of an attacker's position relative to the offside line). We derive an explicit maximum-likelihood rule that determines when the referee's call should stand and when a conflicting VAR measurement should overturn it. Our model treats the true offside margin as fixed and unknown and assumes independent additive Gaussian errors in the human's perception and the AI measurement. When both errors have zero means and known standard deviations $σ_H$ and $σ_{AI}$, respectively, the reversal threshold is $\sqrt{2/π}\,σ^2_{AI}/σ_H$. An onside call is overturned precisely when VAR's measured offside margin exceeds this threshold---not merely because VAR places the attacker beyond the offside line. We also derive the rule when either source has a systematic bias. We prove that the maximum-likelihood estimate of the true margin exists and is unique, while the final binary decision requires only a threshold comparison, without computing that estimate numerically. We illustrate the rule through numerical examples and show how to estimate the error parameters from calibration data. The same analysis applies beyond soccer to binary classification problems in which a human call and an AI measurement satisfy the stated model. This paper is intended for educational purposes: we aim to illustrate how undergraduate students and advanced high school students familiar with basic probability and calculus can investigate research questions in AI and its applications.

cs.AI · cs.LG · math.OC · stat.ML

版本更新 · v2arXiv:2605.14098

Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning

Yu Gu, Zijun Yu, Vahid Partovi Nia, Masoud Asgharian

原文摘要

Chain-of-thought (CoT) reasoning with self-consistency improves performance by aggregating multiple sampled reasoning paths. In this setting, correctness is no longer tied to a single reasoning trace but to the aggregation rule over a pool of candidate paths, making aggregation uncertainty the central challenge. This issue is critical where confidently incorrect answers are far more costly than abstentions. We introduce a conformal procedure for CoT reasoning that directly addresses aggregation uncertainty. Our approach replaces majority voting with weighted score aggregation over reasoning paths and calibrates an abstention rule using conformal risk control. This approach leads to finite-sample guarantees on the confident-error rate--the probability that the system answers and is wrong. We further identify score separability as the key condition under which abstention provably improves selective accuracy, and derive closed-form expressions that predict accuracy gains from calibration data alone. The method is fully inference-time, and requires no retraining. Across four benchmarks, four open-source models, and three score classes, realized confident-error rates are consistent with the prescribed targets up to calibration-split and test-set variability. Our method achieves $90.1\%$ selective accuracy on GSM8K by abstaining on fewer than $5\%$ of problems, compared with $82.3\%$ accuracy under majority voting and $87.4\%$ under score-weighted voting without abstention, while conformal risk control keeps the expected rate of confidently wrong answers below $10\%$.

stat.ML · cs.CL · cs.LG

版本更新 · v2arXiv:2605.08811

Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity

Zhongjie Shi, Wenjing Liao

原文摘要

This paper investigates the learning theory of Transformer networks for regression tasks on the compact Euclidean domain $[0,1]^d$ and $d$-dimensional compact Riemannian manifolds. We propose a novel constructive approximation framework for Transformers that builds local approximations of the target function and aggregates them into a global approximation via softmax partition of unity (POU). This approach leverages the attention mechanism to achieve spatial localization through affine transformations of the input. The softmax activation plays a crucial role in aggregating local approximations to a global output. From an approximation perspective, we prove that a dense Transformer equipped with only two encoder blocks and standard single-hidden-layer point-wise feed-forward networks can achieve a uniform $\varepsilon$-approximation error for $α$-Hölder continuous functions with $α\in (0,1]$ using $\mathcal{O}(\varepsilon^{-d/α})$ total parameters. Building upon this approximation guarantee, we establish a near minimax-optimal generalization error bound of order $\mathcal{O}\big(n^{-\frac{2α}{2α+d}} \log n\big)$ for the empirical risk minimizer, where $n$ is the training data size. The Transformer architecture studied in this paper uses native softmax attention and sinusoidal positional encodings, with a fixed depth of two encoder blocks. Finally, numerical experiments reveal spatially localized, Gaussian-like attention weights that emerge during training, resembling our Softmax POU weights.

stat.ML · cs.LG

版本更新 · v2arXiv:2602.13804

Attention in Constant Time: Vashista Sparse Attention for Long-Context Decoding with Exponential Guarantees

Vashista Nobaub

原文摘要

Large language models spend most of their inference cost on attention over long contexts, yet empirical behavior suggests that only a small subset of tokens meaningfully contributes to each query. We formalize this phenomenon by modeling attention as a projection onto the convex hull of key vectors and analyzing its entropic (softmax-like) relaxation. Our main theoretical contribution is a face-stability theorem showing that, under a strict complementarity margin (a support gap (Δ) certified by KKT multipliers), entropic attention concentrates on a constant-size active face: the total mass assigned to inactive tokens decays exponentially as (\exp(-Ω(Δ/\varepsilon))), while the error on the active face scales linearly in the temperature/regularization parameter (\varepsilon). This yields a practical criterion for when sparse long-context decoding is safe and provides a principled knob to trade accuracy for compute. Building on these guarantees, we introduce Vashista Sparse Attention, a drop-in mechanism that maintains a small candidate set per query through a paging-style context selection strategy compatible with modern inference stacks. Across long-context evaluations, we observe stable constant-size effective support, strong wall-clock speedups, and minimal quality degradation in the regimes predicted by the support-gap diagnostics. Finally, we discuss deployment implications for privacy-sensitive and air-gapped settings, where interchangeable attention modules enable predictable latency and cost without external retrieval dependencies.

cs.AI · cs.LG · stat.ML

版本更新 · v3arXiv:2605.09214

Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability

Qingyue Zhao, Kaixuan Ji, Heyang Zhao, Quanquan Gu

原文摘要

\emph{Kullback-Leibler} (KL) regularization is ubiquitous in reinforcement learning algorithms in the form of \emph{reverse} or \emph{forward} KL. Recent studies have demonstrated $ε^{-1}$-type fast rates for decision making under reverse KL regularization, in contrast to the standard $ε^{-2}$-type sample complexity. However, for forward-KL-regularized objectives, existing statistical analyses are either not applicable or result in $\tilde{O}(ε^{-2})$ slow rates. We take the first step towards addressing this problem via a streamlined analysis of forward-KL-regularized offline CBs. We give the first $\tilde{O}(ε^{-1})$ upper bounds in tabular and general function approximation settings, both under notions of \emph{single-policy concentrability}. In particular, our convex-analytical pipeline unifies these settings by exploiting the pessimism principle in a novel way and completely bypasses the proof routines in previous works based on the mean value theorem, which might be of independent interest. Moreover, we provide rate-optimal lower bounds, manifesting the tightness of our upper bounds in terms of statistical rates. Our lower bounds also demonstrate that the forward-KL-regularized sample complexity recovers the unregularized slow rate in the low-regularization regime, similarly to the reverse-KL regularization.

cs.LG · cs.AI · cs.IT · math.ST · stat.ML

版本更新 · v5arXiv:2605.27991

Gradient-Flow Optimization as Dynamic Random-Effects Inference: Testing and Early Stopping with Applications to Deep Learning

Minhao Yao, Ruoyu Wang, Xihong Lin, Lin Liu, Zhonghua Liu

原文摘要

Gradient-flow optimization is usually viewed as an algorithmic procedure for minimizing empirical loss, with training duration selected by validation or heuristic early stopping rules. We develop a statistical inference framework for gradient-flow training. We show that whenever fitted values evolve through a time-invariant positive semidefinite training operator, the output at each time is equivalent to the best linear unbiased predictor under a corresponding random-effects model. Training time then becomes a variance-component parameter governing variance reallocation from residual noise to structured signal. This turns two training decisions into inferential problems: whether training is needed becomes a variance-component test for signal beyond initialization, and how long to train becomes restricted maximum likelihood (REML) estimation of the training-time variance component. We show that the REML-guided early stopping rule selects the time at which optimized spectral losses become decorrelated from the training-operator eigenvalues. The asymptotic prediction optimality of the REML-guided early stopping time is established for fixed-design in-sample risk and random-design out-of-sample risk. Deep learning models in fixed-kernel gradient regimes provide canonical instantiations for our results. Numerical experiments and a UK Biobank proteomics application show competitive accuracy of the REML-guided early stopping time with reduced reliance on validation splits and repeated checkpoint evaluation.

stat.ML · cs.LG

新论文 · v1arXiv:2611.11744

Breaking the $\sqrt{d}$ Communication Barrier in Federated Sampling with Adaptive Hamiltonian Monte Carlo

Jiajun Liang, Linxuan Wang, Guang Lin, Qifan Song

原文摘要

This work proposes the Adaptive Federated Hamiltonian Monte Carlo (AFHMC) for Bayesian federated learning (FL) tasks. The prior work on HMC applications for FL achieves $O(\sqrt{d}/ε)$ communication cost under log-convex distributions, already demonstrating its advantage over the Langevin dynamic-based counterpart. Based on a refined analysis, this paper further reveals the benefit of HMC for the FL regime by incorporating a control mechanism for local node shifts. We prove that AFHMC at most requires $O(\{d/ε^2\}^{1/3})$ communication cost up to a logarithmic term, which is significantly better than the existing results. Moreover, the proposed method adapts to client heterogeneity. Under low-heterogeneity settings (i.e., the target sampling precision $ε^2$ is higher than the heterogeneity level), the communication rate of AFHMC can be as good as $O(\log(d/ε^2))$, resembling the communication costs for FL optimization tools. To the best of our knowledge, this provides the first logarithmic communication complexity result for federated sampling, even though it requires low heterogeneity. Numerically, we also justify AFHMC's advantage under the low-heterogeneity regime. Our code is available at https://github.com/laaaarrywang/Adaptive-Federated-HMC

stat.ML · cs.LG

版本更新 · v3arXiv:2310.11531

Informed Posterior Sampling: More Efficient Online Learning with Few Offline Demonstrations in Average-Reward MDPs

Dengwang Tang, Dongze Ye, Botao Hao, Rahul Jain, Zheng Wen

原文摘要

We study efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with an unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret than if it does not do that. We establish an upper bound on regret of the exact informed PSRL (iPSRL) algorithm that scales as $\widetilde{\mathcal{O}}(\sqrt{T})$. This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the informed RLSVI (iRLSVI) algorithm to efficiently approximate the iPSRL algorithm. Empirical results show that small offline expert datasets are sufficient for iRLSVI to outperform online-only algorithms on hard exploration domains.

cs.LG · cs.AI · eess.SY · stat.ML

版本更新 · v3arXiv:2605.24274

A lift for input-convex neural net training

Ali Siahkoohi

原文摘要

Input-convex neural nets parametrize the convex potentials of density models and transport maps, and their convexity requires the inter-layer weights to be non-negative. Projected gradient descent enforces this by projecting after each step, and due to mini-batch noise the boundary is re-crossed indefinitely, which leads to an active set the projection never identifies. The differentiable alternative, direct softplus, optimizes a free latent weight through a softplus positivity map. Where that latent weight is well below zero, the derivative of the map is exponentially small and scales down both the gradient and its noise. We call this region the shoulder, and a weight that reaches it therefore barely moves and stays for an exponentially long time. We propose the lift, which replaces the free latent weight by a learnable slack plus the output of a hypernetwork that reads a permutation-invariant summary of the training batch. The latent weight then varies with the same batch that forms the gradient, so the two are coupled. We show that this coupling adds a term to the variance of the update at first order, and that the slack, the batch dependence, and the shared batch are each needed for it. Where the coupling aligns positively with the loss curvature, a weight on the shoulder crosses the barriers of the loss sooner under the lift than under direct softplus, also under preconditioned stochastic gradient descent. We compare the lift with the two existing methods on several applications. Where a constrained weight of direct softplus reaches the shoulder, the lift fits the target more closely than direct softplus and reaches the same reconstruction about three times sooner. Where almost none reaches it, the methods agree.

cs.LG · stat.ML

新论文 · v1arXiv:2611.11675

Query- and Candidate-Aware Long User Sequence Modeling for Sponsored Product Ranking in Search

Dongyue Xie, Sergey Kolbin, Manavender Malgireddy, Iurii Shcherbak, Weixing Tang, Arushi Jain, Wanchen Shao, Kurt Zimmer 等 10 位作者

原文摘要

User sequence modeling is widely used in recommendation and advertising. Search ranking presents an additional challenge: the current query expresses immediate intent, while a candidate product determines which parts of a user's long-term history are relevant. We present a production sequence framework for sponsored-product ranking that augments recent behavior with two retrieval channels over long-term history: candidate-nearest-neighbor retrieval and query-nearest-neighbor retrieval. The selected actions are encoded together with dedicated query and candidate context tokens, while candidate-aware early fusion injects the candidate into every history token. Offline ablations show that dual retrieval improves major ranking metrics relative to the production baseline. In an online A/B test on sponsored-product search traffic, the model increased CTR, attributed conversion rate, and orders per impression.

stat.ML · cs.LG

新论文 · v1arXiv:2611.11667

Nonasymptotic causal structure learning in generalized linear DAGs with continuous optimization

Chang Deng, Inbeom Lee, Bryon Aragam

原文摘要

We study the problem of learning directed acyclic graphs from observational data in nonlinear models and provide finite-sample, nonasymptotic statistical guarantees in these models. Although widely studied, statistical guarantees for gradient-based methods in nonlinear models are missing from the literature. Under a generalized linear model for each node, we analyze a global nonsmooth objective combining an exact acyclicity constraint with a quasi-minimax concave penalty, which can be optimized via standard gradient-based solvers. We establish finite-sample guarantees under this model, including the most widely used logistic and Poisson models. Experiments corroborate structure recovery accuracy and computational performance, including both standardized data and negative controls as baselines.

stat.ME · cs.LG · math.ST · stat.ML

新论文 · v1arXiv:2611.11649

A Trust Region Approach for Learning Schrödinger Bridges

Takeshi Koshizuka, Denis Blessing, Max Zimmer, Sebastian Pokutta, Lorenz Richter

原文摘要

We study the numerical approximation of Schrödinger Bridges in the setting where samples from the target distribution are unavailable and only an unnormalized target density is given. In this regime, the widely used iterative proportional fitting (IPF) procedure (and its variants) often exhibits severe numerical instabilities, stemming from ill-conditioned underlying control problems and from the fact that the optimal solution may lie far from the initialization. To address these issues, we propose a trust region variant of IPF that constrains successive updates to remain at a fixed Kullback-Leibler divergence from the current iterate. This restriction significantly improves numerical stability. Moreover, we show that the resulting method admits an interpretation as a mirror descent scheme with an adaptively chosen step size. We demonstrate the effectiveness of our approach on several challenging sampling tasks.

cs.LG · stat.ML

版本更新 · v2arXiv:2605.14343

Nearest-Neighbor Radii under Dependent Sampling

Yuanyuan Gao, Yilong Hou, Zhexiao Lin

原文摘要

Nearest-neighbor methods are fundamental to classical and modern machine learning, yet their geometric properties are typically analyzed under independent sampling. In this paper, we study the nearest-neighbor radii under dependent sampling. We consider strong mixing dependent observations and ask whether dependence changes the scale of nearest-neighbor neighborhoods. We establish distribution-free almost sure convergence under polynomial mixing and sharp non-asymptotic moment bounds under geometric mixing. The moment bounds depend on the local intrinsic dimension rather than the ambient dimension, making the results applicable to high-dimensional data supported on lower-dimensional manifolds. Synthetic experiments and real-world time-series benchmarks illustrate the theory, showing that nearest-neighbor geometry remains informative under dependent sampling.

cs.LG · math.ST · stat.ML

版本更新 · v3arXiv:2610.10600

The optimal information complexity of VC learning

Steve Hanneke, Juexiao Wang

原文摘要

Steinke and Zakynthinou(2020) introduces the Conditional Mutual Information (CMI) framework of analyzing the information complexity of learning algorithms based on algorithm-dependent information-theoretic quantities. We study one of these quantities, the evaluated Conditional Mutual Information (eCMI). It has been an interesting question whether the optimal PAC guarantee for VC classes can be recovered from the algorithm-dependent analyses via CMI. And we show that it is possible to recover this guarantee by constructing a learning algorithm whose eCMI is of order O(d) in the realizable case, where d is the VC-dimension of the concept class. Specially, our algorithm is a randomized Majority-of-5 base learners with optimal in-expectation generalization guarantee.

stat.ML · cs.IT · cs.LG · math.ST

版本更新 · v3arXiv:2410.04655

Graph Fourier Neural Kernels: Learning Full PDE Trajectories Across Geometries and Anisotropic Parameter Fields

Shane E. Loeffler, Zan Ahmad, Shiyi Chen, Syed Yusuf Ali, Carolyna Yamamoto, Kensuke Sakata, Dan M. Popescu, Alana Yee 等 11 位作者

原文摘要

High-fidelity simulations of time-dependent nonlinear partial differential equations (PDEs) are expensive to repeat across families of domains and parameter fields, a cost that recurs in design optimization, uncertainty quantification, and computational medicine. Most neural-operator surrogates require each domain to be mapped onto a common grid or reference shape, and few predict the full time course. We introduce Graph Fourier Neural Kernels (G-FuNK), a neural operator for second-order semilinear PDEs with a diffusive principal part that transfers across geometries and anisotropic diffusion tensors without coordinate maps or diffeomorphisms. A weighted graph and its Laplacian supply a spectral basis that encodes geometry and anisotropy. Shared layers act in this basis, at a cost linear in mesh size after one partial eigendecomposition per geometry, and a neural ODE integrator propagates the learned generator to full spatiotemporal trajectories on unseen domains. Across anisotropic heat flow, stiff reaction-diffusion, FitzHugh-Nagumo, and patient-specific left-atrial electrophysiology on CT-derived surfaces, G-FuNK reproduces held-out trajectories with relative $L^2$ error between 0.046 and 0.160. Its accuracy is comparable to that of a geometry-aware Fourier neural operator on regular domains, and its predictions are unchanged when a rectangular domain is rotated. On left atria, whose venous and valvular openings prevent mapping onto a reference shape, it predicts an 85-ms trajectory in under half a second, against about 5 minutes for finite elements, with a mean activation-time error of 1.73 ms. Accuracy depends on graph construction and is highest when the learned generator is integrated at its training time step.

cs.LG · cs.AI · math.SP · stat.ME · stat.ML

版本更新 · v2arXiv:2605.29351

Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics

Matthew Smart, Soumya Ganguly, Nilava Metya, Alexandre V. Morozov, Anirvan M. Sengupta

原文摘要

We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-weighted posterior mean with respect to the empirical distribution defined by the context. Depth refines this distribution through particle dynamics (Stage 1), while a long-range skip-connection carries the noisy input as a query for posterior inference (Stage 2), revealing distinct statistical roles for depth and attention residuals. The framework isolates a minimal setting in which the context itself induces a depth-dependent energy landscape governing in-context inference. We show that effective denoising can emerge without an explicit noise schedule: a fixed kernel bandwidth and finite integration horizon suffice, yielding a principled depth-noise relationship. We further establish a posterior-mean recovery guarantee for Gaussian priors, showing the empirical estimator converges to the Bayes-optimal predictor under sequential asymptotic conditions. Connecting these dynamics to reverse-diffusion limits, our results provide a statistical interpretation of attention as in-context inference via sample-based posterior estimation, without explicit density modeling.

cs.LG · math.DS · stat.ML

版本更新 · v2arXiv:2607.26346

Toward a Unified Statistical Theory of Multi-View Pretraining for Neural Knowledge Graph Learning

Jifan Zhang, Miklos Racz, Suqi Liu

原文摘要

Knowledge graph learning offers a powerful framework for representing and inferring structured knowledge, with broad practical applications. Two obstacles limit current methods: labeled triples are scarce for each entity and relation, which makes expressive models hard to train, and scoring functions are designed ad hoc, which limits their generalizability and leaves them without theoretical grounding. We address both obstacles with a theoretically grounded framework, with end-to-end guarantees, that extends and subsumes existing methods. The framework proceeds in two stages: multi-view unsupervised pretraining that integrates heterogeneous sources of entity side information into a shared representation, followed by supervised learning across multiple relation types. We establish a unified nonasymptotic risk bound that combines the subspace recovery error of pretraining and the sample complexity of the labeled data in a single guarantee, quantifying formally how large-scale unlabeled data from multiple views benefits downstream knowledge prediction. Synthetic experiments validate each component of the theory, and experiments on large-scale real-world knowledge graph benchmarks confirm the effectiveness of our approach.

math.ST · cs.SI · stat.ME · stat.ML