We study online Wasserstein distributionally robust optimization (DRO), where the ambiguity set is a Wasserstein ball centered at the empirical distribution and evolves with each observation. We formulate each round as a zero-sum game and develop an online distributional best-response algorithm that requires only one projected subgradient step per iteration. Combining regret analysis with measure concentration, we establish explicit, instance-dependent, and high-probability convergence rates to the optimal offline DRO value.
Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the \emph{accumulated effective learning rate}, $C_T=\sum_{t<T}η_t/\|W_t\|_F$, where $η_t$ is the nominal learning rate and $η_t/\|W_t\|_F$ is the effective learning rate relative to the weight norm. For bounded update directions, accumulation over any training interval bounds relative weight displacement. We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress, whereas MuonW and MuonH guarantee at least linear accumulation under the stated conditions. Going beyond bounded-norm control, we show that sufficiently slow norm growth guarantees faster accumulation than the logarithmic worst case, opening up less restrictive optimizer designs. Based on this insight, we propose MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. With bounded update directions and a constant nominal learning rate, we prove that the weight norm grows at most proportionally to $\sqrt{t}$ after $t$ steps. Consequently, MuonGS guarantees $C_T=Ω(\sqrt{T})$, with accumulation sustained even in later training. Further, for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee. In pretraining experiments with 160M- and 400M-parameter dense models as well as a 135M active- and 1B total- parameter MoE model, MuonGS achieves lower validation perplexity than MuonH at long training horizons. In particular, for the dense model it achieves an estimated token-equivalent speedup up to about $15\%$ at matched validation loss to MuonH.
Ali Ghasemzadeh, Erfan Babakhani, Armin Khosravi, Pariya Ghasemzadeh, Yong-Hong Kuo, Seyedali Mirjalili, Amir H. Gandomi
Many real-world optimization tasks are governed not only by what should be optimized, but also by what is allowed: a solution must first satisfy physical, operational, safety, budgetary, or structural requirements before its objective value becomes meaningful. Yet benchmark design for constrained continuous optimization often treats these restrictions as fixed properties of predefined test instances. Existing generators and benchmark suites provide essential foundations for reproducible comparison, but they either focus primarily on configurable objective landscapes or expose constraint properties within limited, predefined regimes, making it difficult to ask targeted questions about constraint-induced difficulty: whether an optimizer struggles because feasible regions are small, boundaries are restrictive, variables are tightly coupled, or constraint information is poorly scaled. We introduce GNBG-C, a Generalized Numerical Benchmark Generator for Constrained Continuous Optimization. The central idea is to treat the feasible region itself as a configurable object rather than as an inseparable attachment to a fixed test function. By generating controlled constraint regimes around the known GNBG global minimizer, the generator enables systematic studies in which feasibility pressure can be varied while preserving that point as a global minimizer of the constrained problem. This allows researchers to examine how optimizers discover feasibility, handle constraint pressure, and improve objective quality under diverse and controllable feasibility conditions. GNBG-C therefore provides a flexible and auditable framework for probing the strengths and weaknesses of constrained continuous optimizers under targeted forms of feasibility-side difficulty.
Zhenyu Wang, Molei Liu, Jing Lei, Francis Bach, Zijian Guo
When synthesizing multi-source high-dimensional data, a key objective is to extract low-dimensional representations that effectively approximate the original features across different sources. Such representations facilitate the discovery of transferable structures and help mitigate systematic biases such as batch effects. We introduce Stable Principal Component Analysis (StablePCA), a distributionally robust framework for constructing stable latent representations by maximizing the worst-case explained variance over multiple sources. A primary challenge in extending classical PCA to the multi-source setting lies in the nonconvex rank constraint, which renders the StablePCA formulation a nonconvex optimization problem. To overcome this challenge, we conduct a convex relaxation of StablePCA and develop an efficient Mirror-Prox algorithm to solve the relaxed problem, with global convergence guarantees. Since the relaxed problem generally differs from the original formulation, we further introduce a data-dependent certificate to assess how well the algorithm solves the original nonconvex problem and establish the condition under which the relaxation is tight. Finally, we explore alternative distributionally robust formulations of multi-source PCA based on different loss functions.
We present sparse dictionary differentiable predictive control (SD-DPC), a framework for learning sparse, interpretable feedback policies for nonlinear systems from data. A prediction model is first identified by rollout-based sparse identification of nonlinear dynamics (SINDy), building on gradient-based and multistep formulations. The policy is then parameterized as a sparse combination of dictionary functions and trained by differentiating a constrained finite-horizon predictive-control objective through this model, so that its terms are selected by closed-loop performance rather than by imitating a previously trained controller. The result is an explicit feedback law with only a handful of terms. Across three benchmark control problems, SD-DPC satisfies the constraints in almost all test scenarios, outperforms a policy distilled onto the same terms by up to an order of magnitude, and requires orders of magnitude less memory and online computation than an optimization benchmark, while admitting explicit sensitivity bounds.
Elliot Paquette, Noah Marshall, Lucas Benigni, Guangyuan Wang, Atish Agarwala, Courtney Paquette
Muon and related spectral optimizers have recently demonstrated strong empirical performance as scalable stochastic methods, often outperforming Adam. Yet their behavior remains poorly understood. We analyze stochastic Muon, which orthogonalizes an exponential moving average of minibatch gradients, on a high-dimensional matrix-valued least squares problem, and we compare it with stochastic SignSGD, a simple baseline comparable to Adam. We derive explicit deterministic dynamics that provide a tractable framework for studying learning behavior. We show that Muon has non-linear sensitivity to the effective batch $B_{\rm eff}=B(1+β)/(1-β)$. With exact whitening and small $B_{\rm eff}$ Muon fails to resolve smaller eigenmodes, slowing down convergence. Momentum and finite Newton--Shulz iterations can act as a ridge which lessen this effect. This behavior is in contrast with SignSGD which has no transition, leading to different optimal learning rates, but with matching convergence rate up to a constant factor in the isotropic setting. For anisotropic data, large batch setting Muon performs a square-root preconditioning with respect to the data covariance spectrum while SignSGD provides no preconditioning for generic covariances. We analyze time to target loss in a power law covariance model and derive a phase diagram in the data exponent $α$ and target exponent $ζ$. Muon is favored for steep covariance spectra with rich targets ($α>2$, $ζ<α+1$), SignSGD is favored for flat spectra ($α<1$), and in other regions the winner depends on how the target loss scales with dimension. Our results shed light on why replacing Adam with Muon is not trivial even from a pure optimization perspective.
We study over-the-air computation of functions, such as empirical variance, formed by combining two separately postprocessed sums of locally transformed observations. We consider continuous encoders subject to pointwise peak-power constraints over a single use of a complex multiple-access channel. With four or more transmitters, we identify conditions under which exact noiseless recovery forces every continuous encoder collection into a common affine structure that preserves both sums, reducing infinite-dimensional encoder design to selecting a finite set of coefficients. To quantify resilience against channel noise, we introduce a robustness score that measures the separation of noiseless received aggregates relative to variations in the desired function value. Within this characterized encoder family, we formulate robustness maximization as an exact convex semi-infinite program, eliminating rank constraints without a relaxation gap, and construct a decoder that attains the minimum uniform noise-amplification constant. For several function classes, including empirical variance, we derive closed-form robustness-optimal encoders and scores. Under mild regularity conditions, we also establish sharp limits on the Gaussian mean squared error at high signal-to-noise ratios and show how using the input distribution reduces average error at the expense of uniform robustness.
Classical pooled-testing rules choose pool sizes to reduce the number of tests. In practice, however, samples arrive over time, pools take time to form, testing capacity is finite, and positive pools create bursts of follow-up work. These operational features can make the test-efficient pool size far from the delay-efficient one. We formulate pooled testing with stochastic arrivals and finite testing capacity as a queueing system, and derive exact performance representations and asymptotic scaling laws for fixed-batch Dorfman testing and a dynamic two-threshold policy. The analysis reveals three distinct design regimes. When test usage is the main concern, the classical Dorfman rule is recovered and the optimal pool size scales with the inverse square root of prevalence. When delay accumulated while the laboratory is busy testing another batch matters, positive pools create service-time variability that grows more rapidly with pool size than the mean retesting burden, and the optimal scale shifts to the inverse cube root of prevalence. When waiting to assemble a pool is itself costly, the optimal fixed pool size remains bounded even as prevalence becomes very small. Under severe capacity scarcity, congestion remains relevant at the classical Dorfman scale: the square-root law survives, but with a capacity-dependent constant. Dynamic pooling separates two operational decisions that fixed pool sizes conflate. A lower threshold determines when routine testing should begin, while an upper threshold limits pooling when backlog is high. The lower threshold follows the cube-root scale, whereas the upper threshold primarily acts as congestion control. The results show how prevalence, capacity, and delay jointly determine pooling decisions and explain why operationally optimal pools can be much smaller than those suggested by test-efficiency calculations alone.
The Low Order-Value Optimization (LOVO) problem involves minimizing the minimum among a finite number of function values within a feasible set. LOVO has several practical applications such as robust parameter estimation, protein alignment, portfolio optimization, among others. In this work, we are interested in the constrained nonlinear optimization LOVO problem of minimizing the minimum between a finite number of function values subject to a nonempty closed convex set where each function is a black-box and continuously differentiable, but the derivatives are not available. We develop the first derivative-free trust-region algorithm for constrained LOVO problems with convergence to weakly critical points. Under suitable conditions, we establish the global convergence of the algorithm and also its worst-case iteration complexity analysis. An initial open-source implementation using only linear interpolation models is developed. Extensive numerical experiments and comparison with existing alternatives show the properties and the efficiency of the proposed approach when solving LOVO problems.
Robust optimization models that allow decisions to adapt to newly revealed information remain computationally challenging and are often limited to two decision stages or are addressed heuristically or via approximation schemes. This challenge is further amplified when uncertainty sets depend on earlier decisions. Conventional approaches to such problems often require problem-specific reformulations and tailored algorithms, highlighting the need for an exact model-and-run approach. We address this gap by presenting an algorithmic framework for solving multistage robust discrete linear optimization problems. This exact approach adapts and integrates ideas from game-tree search, quantified conflict-driven clause learning, and integer linear programming to explore the minimax tree and is implemented in the open-source solver Yasol. The solver supports discrete decision variables across multiple stages, discrete uncertainty sets with polyhedral or decision-dependent structure, and mixed-integer recourse in the final stage. It can be used as a standalone executable or through a C++ library interface, allowing it to serve as a reference solver for exact methods in multistage robust discrete linear optimization. In an extensive computational study, we demonstrate the capabilities of our solver. The results show competitive performance against a reference approach from the literature for a protection-interdiction problem, as well as strong performance on bilevel benchmark instances. For a multistage cutting-stock problem, we demonstrate the substantial loss in solution quality that can arise from neglecting the multistage structure and show that our solver significantly outperforms decision-rule approximations in terms of computational efficiency while certifying the true multistage optimum.
Mohammad Al-Jarrah, Michele Martino, Marcus Yim, Bamdad Hosseini, Amirhossein Taghvaei
We present Conditional Wasserstein Autoencoders (CWAEs), a framework for conditional simulation that exploits low-dimensional structure in both the conditioned and the conditioning variables. The key idea is to modify a Wasserstein autoencoder to use a (block-) triangular decoder and impose an appropriate independence assumption on the latent variables. We show that the resulting model gives an autoencoder that can exploit low-dimensional structure while simultaneously the decoder can be used for conditional simulation. We explore various theoretical properties of CWAEs, including their connections to conditional optimal transport (OT) problems. We also present alternative formulations that lead to three architectural variants forming the foundation of our algorithms. We present a series of numerical experiments that demonstrate that our different CWAE variants achieve substantial reductions in approximation error relative to the low-rank ensemble Kalman filter (LREnKF), particularly in problems where the support of the conditional measures is truly low-dimensional.
High-frequency control problems are often studied through time discretization, but refining the timestep $δ$ increases the discrete horizon $H=T/δ$ and can make standard finite-horizon regret guarantees deteriorate even when the physical horizon $T$ is fixed. We study episodic value-based reinforcement learning for Euler discretizations of controlled diffusions. The performance is measured as the cumulative regret over $K$ episodes relative to the optimal policy of the corresponding Euler model. Under stability and regularity conditions on value function gradients, Bellman residual derivatives, and fitted critics, we prove an $\mathcal{O}(\log K)$ regret bound with constants uniform in $δ$. For dynamics with drift linear in unknown parameters and unknown constant diffusion covariance, we verify localized versions of these conditions for a constrained, model-based controlled-diffusion fitted-$Q$ algorithm (CD-FQ) and obtain high-probability $\mathcal{O}(\log K\log\log K)$ regret for fixed and finite $T$, again with constants uniform in $δ$.
Jia-Nan Wang, Zixun Huang, Kairui Li, Zilin Wang, Lei Wu
We study when and how momentum improves large-batch training in the one-pass regime, using power-law kernel regression as a tractable setting. We first characterize stochastic stability through the \emph{critical learning rate}, defined as the largest learning rate for stable training, and obtain $η_{\mathrm{SGD}}^{\mathrm{crit}}\eqsim 1$, $η_{\mathrm{Polyak}}^{\mathrm{crit}}\eqsim \min\{1,B(1-β)\}$, and $η_{\mathrm{Nesterov}}^{\mathrm{crit}}\eqsim \min\{1,B^q(1-β)\}$, where $B$ is the batch size, $β\in[0,1)$ is the momentum factor, and \(q>1\) is the capacity exponent governing spectral decay. Within the stable region, we derive scaling laws for the full risk dynamics, capturing the progression from an early transient through power-law decay to a noise floor. We then optimize the final-step risk under a fixed data budget, obtaining a three-regime phase diagram that reveals how the benefits of momentum change with batch size. Notably, Polyak enlarges the critical batch size, the largest batch size preserving the optimal small-batch data-scaling exponent, thereby allowing fewer sequential updates without sacrificing data efficiency. In contrast, Nesterov improves the optimal data-scaling exponent in the large-batch regime through look-ahead-induced suppression of noise amplification. Experiments on PLK regression quantitatively support the theoretical scaling predictions, while neural-network experiments provide qualitative evidence for the predicted batch-size dependence and stability behavior.
Mateo Díaz, Nicolás García Trillos, Daniel López-Castaño, Congwei Yang
Dimension-reduction methods such as PCA identify a subspace capturing most of the variation in the data, but not a distinguished basis within it. Selecting a basis in which the samples are sparsely represented is a recurring task in unsupervised learning, addressed by methods in factor analysis, dictionary learning, and subspace clustering. We pose this task as Wasserstein approximation of the data distribution by measures supported on the axes of an orthogonal frame, which, in turn, reduces to a nonsmooth optimization problem over the orthogonal group $\mathrm{O}(d)$. We show that, under mild assumptions, both empirical risk minimizers and a locally initialized stochastic gradient method recover the population-optimal frame at the near-parametric rate $\widetilde{O}_d(n^{-1/2})$, where $n$ is the sample size. The main difficulty is that, although the population objective is smooth, the empirical objective is not-this mismatch precludes standard covering arguments. We overcome this issue by replacing Lipschitz continuity of the empirical gradient with a coarsely Lipschitz bound, a tool that may be of independent interest. Numerical experiments support our theoretical findings.
Yifan Zhao, Qianyang Zhao, Xinyu Guan, kai wei, Yuming Deng
Chunking is a fundamental preprocessing step in retrieval-augmented generation (RAG), yet most existing chunkers rely primarily on heuristics. We introduce the Question--Evidence-Supervised Chunker (QESChunker), to our knowledge the first method for constructing static chunk indices from question--evidence supervision. QESChunker constructs document-specific chunk indices by optimizing over human-annotated or LLM-generated question--evidence pairs, without access to held-out test queries. We formulate non-overlapping and overlapping index construction through a unified question--evidence utility-maximization objective, turning chunking into a supervised combinatorial optimization problem. Under the non-overlap constraint, the objective decomposes over chunks and can be optimized exactly in polynomial time by dynamic programming. When overlap is allowed, the objective becomes monotone submodular; exact optimization is NP-hard, but our greedy achieves a $(1-1/\sqrt{e})$-approximation. Across two self-constructed and three public datasets, both variants outperform existing chunking methods in complete-evidence recall in nearly all evaluated settings, under the same overlap regime and a fixed hybrid retriever. These results recast chunking as supervised, budget-aware index construction for evidence retrieval.
Derivative-free optimization (DFO) addresses problems where derivatives are unavailable or costly to obtain. This setting often arises in simulation-based and experimental applications, where evaluating the black-box objective can itself be costly. Thus, it remains a central challenge in DFO to effectively exploit accumulated observations to select informative evaluation points under a limited budget. In this light, we introduce the NextEval solver, an agent-based DFO solver, which integrates the capabilities of a large language model (LLM) to guide the sequential selection of evaluation points. Specifically, we establish a system that combines an LLM agent with a numerical environment. The environment provides tools for analyzing past evaluations and executing numerical methods, while maintaining a shared evaluation history and method-specific state to support continuation across method switches. Using observations and tool feedback, the LLM agent selects tools, configures their parameters, and allocates the evaluation budget, while numerical methods compute and evaluate candidate points. Building on this system, we also introduce NextEval Bench, a benchmark protocol for comparing agent configurations on shared DFO tasks using the same numerical tools under matched evaluation budgets and information constraints. Numerical results on low-dimensional DFO problems demonstrate that the NextEval solver outperforms the state-of-the-art DFO baselines in function-evaluation efficiency.
Mingyuan Xu, Zongqi Xia, Tianxi Cai, Doudou Zhou, Nian Si
Learning a common policy from multiple source environments requires accounting for cross-source heterogeneity, and the offline setting introduces an additional challenge of limited data coverage. We formulate this problem as a distributionally robust Markov decision process under a shared linear MDP representation with source-specific transition and reward components. The uncertainty set is constructed through feature-wise mixtures of these components, yielding a $d$-rectangular structure that captures cross-source heterogeneity and admits a robust Bellman recursion. Building on this formulation, we develop Group-Robust Pessimistic Value Iteration (GR-PEVI), an offline policy-learning algorithm that combines feature-wise worst-case aggregation with a data-dependent pessimism penalty. Under a robust partial coverage condition, we establish finite-sample suboptimality guarantees for GR-PEVI and extend the analysis to clustered sources, characterizing the trade-off between the statistical gain from pooling similar sources and the approximation error induced by within-cluster heterogeneity. Simulation studies demonstrate the effectiveness of GR-PEVI, with empirical behavior consistent with the theoretical analysis. We apply GR-PEVI to a multi-source electronic health record study of multiple sclerosis treatment decisions to illustrate its use in a clinical setting.
Valentin Leplat, Daniil Merkulov, Aleksandr Katrutsa, Daniel Bershatsky, Olga Tsymboi, Ivan Oseledets
Stochastic first-order methods are widely used for large-scale optimization and machine learning, where their performance often depends on the learning rate. We study NAG-GS, a stochastic momentum method based on a semi-implicit Gauss--Seidel discretization of a Nesterov-type flow. For strongly convex quadratics, we determine the exact range of stable step sizes for fixed damping and the step size that minimizes the worst-case spectral radius. When the damping equals the strong-convexity parameter, the method recovers the accelerated asymptotic factor $(\sqrtκ-1)/(\sqrtκ+1)$, where $κ$ is the condition number. We also prove that the quadratic iteration converges linearly with dynamic damping when its limiting fixed-damping iteration is stable. For general strongly convex, possibly nonsmooth objectives, we establish an $O(1/k)$ bound on the expected objective gap of the last iterate, using a conditionally unbiased stochastic first-order oracle with bounded second moment. The result holds with dynamic damping from any $γ_0>0$. We show that NAG-GS can also be written as a Heavy-Ball iteration with explicitly determined step sizes and momentum coefficients. We test the stochastic result on nonsmooth hinge-loss classification problems, and further evaluate NAG-GS on quadratic problems, logistic regression, convolutional neural networks, RoBERTa, and Vision Transformers. The experiments show that NAG-GS can remain effective at learning rates where standard optimizers become unstable or lose accuracy.
James Hazelden, Laura Driscoll, Eli Shlizerman, Eric Shea-Brown
In training a neural network with gradient descent (GD), each iteration induces a linear operator that governs the first-order updates to a model's internal state variables. We call this operator the Global Empirical Neural Tangent Kernel (GeNTK). In finite-width networks, GeNTK is typically intractable to form, leading prior work to focus on restrictive settings, such as tracking only outputs or infinite-width limits. Here, we bridge this gap by studying the structure of GeNTK across many models. Formulating the model state as the solution to a global implicit constraint, we derive GeNTK as a product of two operators: K, accounting for immediate parameter-to-state interactions, and P, describing internal state-to-state dependencies. For a broad class of "weight-based models", such as RNNs and transformers, we prove a universal Kronecker-core theorem showing that K admits an exact, forward-pass-computable form given by the Gram matrix of weight-site variables. This core gives a measurable state-dependent restriction on Euclidean GD, termed "self-referential bias": current activity filters which adjoint directions can drive a first-order update. For recurrent models (GRUs and RNNs), we examine the spectrum of GeNTK and formalize notions of low-rank, biased learning in terms of reduced spatial or temporal ranks of the full operator. Practically, this explains why some RNN initializations yield better learning, while some plateau, by biasing the GeNTK and thus restricting learning to a small domain of learnable tasks. To demonstrate computational viability, we validate the main theorems in a self-attention transformer and a pretrained Pythia transformer language model with 410 million parameters. Overall, our results expose tractable local rank and mode-selective biases in the finite-width GD geometry and open the door to principled study of the GeNTK, a bridge between theory and practice.
The curse of dimensionality associated with solving the Hamilton-Jacobi-Bellman equation for the value function remains a key obstacle for real-time optimal feedback control of high-dimensional nonlinear dynamical systems. This is especially severe in aerospace applications such as relaminarizing unsteady turbulent flows and recovering from aerodynamic stall where nonlinear dynamics play a significant role. We introduce Balanced Active Subspaces for Optimal Control (BASOC), a model reduction framework which seeks to find directions in state space along which the value function is most sensitive, and the distribution of states exhibits maximum variance along optimal trajectories. BASOC leverages trajectory optimization and the Pontryagin minimum principle to obtain state and co-state (value function gradient) data along locally optimal trajectories for constructing state and gradient covariance matrices. Balancing these matrices provides an oblique projection for approximating the value function, which can also be used to construct reduced-order models and surrogate feedback control laws. We show that for linear-quadratic optimal control problems, BASOC becomes equivalent to a snapshot method for initial condition balancing of the optimally controlled closed-loop realization. The framework is demonstrated on a nonlinear three-dimensional system where POD and balanced truncation are known to perform poorly, and on the nonlinear complex Ginzburg-Landau equation to demonstrate feasibility for larger-scale systems.
Large-scale linear programs with millions of variables and constraints strain time and memory budgets. First-order methods such as the primal-dual hybrid gradient scale to this regime because they avoid matrix factorizations, but they require many iterations. Learned accelerators can reduce these iterations, yet fewer iterations do not guarantee a shorter solve. Learning inference also takes time, and running the accelerator longer trades solver time for inference time. Existing unrolled accelerators handle this trade-off poorly: each layer processes every variable and constraint, the predicted iterates still need solver refinement, and every instance runs the same layers. We therefore ask: What neural accelerator design achieves end-to-end speedups on large-scale linear programs across sizes and target accuracies? We introduce AHEAD, an accelerator that predicts step sizes for first-order linear programming solvers while keeping their update rules. Its weight-shared predictor reads compact solver statistics and rescales the primal and dual step sizes, so each warm-start iteration stays inexpensive and a short warm-start phase hands the solver a warm start. A learned handoff policy sets the number of warm-start iterations for each instance and target accuracy, after which the solver finishes under its own termination criterion. On packing linear programs with up to $10^7$ variables, AHEAD achieves $6.5$--$11.6\times$ end-to-end speedups over Gurobi's first-order solver from a cold start, including on instances of unseen sizes and distributions. The gains extend to PageRank linear programs and hold across difficulty levels and target accuracies.
We propose an optimization-informed deep neural network approach, named iUzawa-Net, aiming for the first solver that enables real-time solutions for a class of nonsmooth optimal control problems of linear partial differential equations (PDEs). The iUzawa-Net unrolls an inexact Uzawa method for saddle point problems, replacing classical preconditioners and PDE solvers with specifically designed learnable neural networks. We prove universal approximation properties and establish the asymptotic $\varepsilon$-optimality for the iUzawa-Net, and validate its promising numerical efficiency through nonsmooth elliptic and parabolic optimal control problems. Our techniques offer a versatile framework for designing and analyzing various optimization-informed deep learning approaches to optimal control and other PDE-constrained optimization problems. The proposed learning-to-control approach synergizes model-based optimization algorithms and data-driven deep learning techniques, inheriting the merits of both methodologies.
Estimation error in machine learning prognostic models can propagate into poor maintenance decisions. We study a data-driven predictive maintenance problem coupling health-condition estimation with a joint ordering-replacement model. We show that the conventional Estimate-Then-Optimize (ETO) approach yields inconsistent, suboptimal decisions for a broad class of such problems. To bridge the gap, we propose an Integrated Estimate-Optimize (IEO) framework that directly targets maintenance cost as the prognostic learning objective, bypassing surrogate statistical accuracy metrics such as Kullback--Leibler divergence or mean squared error. Theoretically, we prove out-of-sample performance guarantees of the IEO framework under certain conditions, including i.i.d. and sliding-window data-generating processes. Computationally, we develop a training procedure combining Monte Carlo dropout, central-difference gradient approximation, and an ETO warm-start to handle discrete maintenance decisions and nonlinear costs. Experiments on the four CMAPSS subsets demonstrate that, under the nominal setting, IEO reduces mean maintenance cost by 14.8--22.7% relative to ETO and by 7.2--11.8% relative to a decision-oriented baseline, while also reducing failure rates and cost variation. These improvements are obtained despite higher prediction errors. Increasing the ratio of unplanned-downtime costs to planned-downtime costs makes this tradeoff more pronounced: IEO shifts predictions further toward RUL underestimation, accompanied by greater cost savings and fewer failures. These findings support aligning prognostic learning with maintenance objectives when imperfect predictions must guide decisions under costly unplanned downtime.
Recently, rectified flow has emerged as a fundamental framework for large-scale image generation, powering state-of-the-art systems such as FLUX.1 and Stable Diffusion 3. Despite its remarkable empirical success, the computational and statistical guarantees of iterative rectified flow have remained largely unexplored. We address this problem by studying c-rectified flow, a cost-aware class of rectified flow that projects velocity fields onto a gradient class while preserving endpoint marginals. The ordinary rectified flow can fail to recover the optimal transport coupling: in a Gaussian case study, the iteration converges to the optimal coupling if and only if the source and target covariance matrices commute. More generally, for a Gaussian to log-concave coupling, the ordinary rectified flow can reach a fixed point after one step from independent initialization. In contrast, under suitable regularity and uniform-integrability assumptions, iterative c-rectified flow always converges to the optimal transport coupling. We further establish quantitative one-step contraction and exponential convergence guarantees under projection-stability assumptions for both quadratic and strongly convex displacement costs. Finally, under a Hölder ball assumption, we develop new minimax-optimal score estimation rates and show that, when combined with iterative c-rectified flow, they yield a rate-optimal estimator of the optimal transport for the dimension \(d \ge 3\) and a nearly parametric rate for \(d=1,2\).
We study full-information online linear optimisation on the unit $\ell_p$ ball in $\mathbb{R}^d$, for fixed $2\le p<\infty$ and losses of dual norm at most one. Let $Δ_T$ denote the cumulative loss of the best fixed action above each round's minimum. A follow-the-regularised-leader (FTRL) algorithm attains, on every loss sequence and without knowing $T$ or $Δ_T$, regret $O_p(1+\min\{Δ_T^{1-1/p},\, d^{1/2-1/p}\sqrt{Δ_T}\})$. Matching lower bounds show that this rate is minimax for each fixed $p$, and its constant $(2+o(1))p/\log p$ is optimal as $p\to\infty$ among guarantees uniform in $d$ and $T$. The algorithm uses a bounded radial regulariser and switches at most once to its quadratic augmentation. On some paths where follow-the-leader (FTL) has regret one, its regret grows with $T$. A switching extension keeps the sharp constant and also guarantees $\mathrm{Reg}_T\le 2\,\mathrm{Reg}_T^{\mathrm{FTL}}$. For i.i.d. losses whose mean has no zero coordinates, we compute the exact $\log T$ coefficient of the expected regret, which separates FTL's variance term from the bias of the regulariser.
Nicolás García Trillos, Adam Quinn Jaffe, Bodhisattva Sen
The quantity of interest in the classical Cramér-Rao theory of unbiased estimation (e.g., the Cramér-Rao lower bound, its exact attainment for exponential families, and asymptotic efficiency of maximum likelihood estimation) is the variance, which represents the instability of an estimator when its value is compared to the value for an independently-sampled data set from the same distribution. In this paper we are interested in a quantity which represents the instability of an estimator when its value is compared to the value for an infinitesimal additive perturbation of the original data set; we refer to this as the "sensitivity" of an estimator. The resulting theory of sensitivity is based on the Wasserstein geometry in the same way that the classical theory of variance is based on the Fisher-Rao (equivalently, Hellinger) geometry, and this insight allows us to determine a collection of results which are analogous to the classical case: a Wasserstein-Cramér-Rao lower bound for the sensitivity of any unbiased estimator, a characterization of models in which there exist unbiased estimators achieving the lower bound exactly, and various results on the asymptotic achievement of the lower bound. More precisely, we show that Wasserstein projection estimators (and one-step approximations thereof) are asymptotically sensitivity-efficient under suitable assumptions, and that there exist Hodges-type estimators which exhibit super-sensitivity-efficiency. We use these results to treat many statistical examples, sometimes revealing new optimality properties for existing estimators and other times revealing entirely new estimators.
Zhirayr Tovmasyan, Slavomír Hanzely, Egor Shulgin, Peter Richtárik, Sebastian U. Stich
One of the main challenges of federated learning is heterogeneous data split among the clients, leading to biased local updates called client drift. The SCAFFOLD algorithm (Karimireddy et al., 2020) corrects client drift with control variates, but its relation to the similarity of client data is not yet fully understood: the original similarity-based guarantees cover only quadratics with a stronger second-order similarity assumption, and a follow-up weakly convex analysis (Luo et al., 2025) obtains a $\sqrt{Lδ}$ dependence for $L$-smooth losses with averaged Hessian similarity $δ$. We analyze SCAFFOLD with minibatch-based control variates and establish linear dependence on Hessian similarity beyond quadratics, for both convex and nonconvex objectives. In particular, for convex objectives with exact gradients, $R$ SCAFFOLD rounds with $K$ local steps lead to an objective gap of order $(L/K+δ)D_0^2/R$, where $D_0$ is the initial distance to a minimizer. When $K=Θ(L/δ)$, it improves the minibatch SGD rate by a factor $Θ(L/δ)$. For nonconvex objectives, we prove an analogous stationarity guarantee under uniform Hessian similarity for a client-selection variant. In both settings, we extend these guarantees to stochastic gradients and prove matching lower bounds for the analyzed methods with constant local stepsizes. We also show that, under partial participation, even identical clients need not obtain the full communication benefit of additional local steps.
We formulate flow matching on probability measures over probability measures. Directly from the Wasserstein-on-Wasserstein (WoW) continuity equation, we show that admissible nested outer and inner couplings induce a valid metameasure flow and non-local velocity field. Validity requires neither optimal plans nor a smooth manifold. Practically, this opens up the design space of multisample flow matching: exact Wasserstein plans are distinguished by energy and straightness, while sliced and lazy linear plans yield scalable alternatives. Point-cloud experiments, including controlled Bures-Wasserstein and Euclidean minibatch baselines, demonstrate the resulting geometry-cost trade-off.
We study Multi-Level Aggregation with Maximum Delay (MLA-MD) on finite weighted rooted trees. Each service pays for a rooted subtree and for the maximum waiting time among the requests it clears. For the offline problem, we give an exact polynomial-time algorithm that computes both the optimum and an optimal service schedule. For the online problem, we give a deterministic $2$-competitive algorithm and, against an oblivious adversary, a randomized $e/(e-1)$-competitive algorithm. Both competitive ratios are optimal, even on rooted paths. We also extend the offline result and both online guarantees to service systems with submodular costs.
By virtue of the sheer compute demands of large-scale artificial intelligence (AI) and high-performance computing (HPC), modern multi-core microprocessors suffer from severe power density limits and thermal throttling, entailing an urgent need for principled methods for predictive, physics-informed dynamical thermal management (DTM). Here, we introduce a hardware/software co-design physics-informed PDE-constrained DTM throttling optimal control algorithm, ADACT (Adjoint-Driven Adaptive Co-Design Thermal Control), that minimizes thermal risk and cooling energy overhead while maximizing core execution throughput. The underlying physical temperature field is governed by a two-dimensional nonhomogeneous parabolic reaction-diffusion equation with spatially localized, time-varying power source terms and variable convective cooling subject to nonhomogeneous Dirichlet boundary conditions. We employ a discretization of the continuous state PDE via a Crank-Nicolson finite difference scheme with a 5-point discrete Laplacian operator on a uniform spatial mesh, which preserves unconditional numerical stability and $O(Δx^2 + Δy^2 + Δt^2)$ second-order temporal and spatial accuracy. To efficiently compute exact gradients, we construct and implement a continuous backward-in-time adjoint state method with L-BFGS as our optimization algorithm of choice. The accuracy and second-order convergence rates of both forward state and backward adjoint solvers are quantitatively verified via the Method of Manufactured Solutions (MMS), and gradient accuracy is validated using directional derivative Taylor tests. Numerical experiments demonstrate that ADACT dynamically throttles software power workloads and modulates hardware cooling efforts to strictly uphold physical thermal thresholds while maintaining operational throughput.