论文库

Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because estimation errors in fitted $Q$-functions and occupancy ratios cannot be directly evaluated using held-out transition data. Existing Bellman-residual-based validation methods often require auxiliary regressions, while many minimax-based methods require optimization over separate critic classes. We propose pairwise Bellman validation (PBV), a family of penalized Bellman tournament methods that reduces validation to computing held-out sample averages. PBV constructs critics from pairwise candidate differences, on the log scale for occupancy ratios, and selects the candidate with the smallest worst-case penalized score. This construction approximates oracle critics given by $Q$-function errors and log ratios of candidate to true occupancy ratios, whose Bellman and adjoint Bellman moments control estimation error. Critic penalties account for approximation error under candidate-class misspecification. We establish finite-sample oracle inequalities relative to the best available candidate, even under candidate-class misspecification. For $Q$-function selection, unweighted PBV avoids importance weighting but requires a coercivity condition on a quadratic form induced by the Bellman operator. Importance-weighted PBV (IW-PBV) uses marginalized importance weighting to exploit Bellman contraction under the target-policy occupancy measure, avoiding this condition. For occupancy-ratio selection, adjoint PBV (A-PBV) exploits KL contraction of the adjoint Bellman operator to obtain oracle guarantees under the offline data law. On Hopper and DeepOPE, unweighted PBV is competitive with LSTD-Tournament, often achieving higher accuracy on Hopper, while importance weighting can further improve performance under controlled overlap.

We introduce the Riemannian Lyapunov Framework (RLF), a unified framework that represents first order optimization as a closed loop controlled dynamical system on a Riemannian parameter manifold. Within this framework, optimizers are specified by a quadruple $(M, g, Φ, η)$ consisting of a Riemannian metric, an internal state transition map, a direction field generator, and a lifting parameter. Classical and modern methods including SGD, Adam, AdamW, Lion and Shampoo are recovered as specific instantiations of this quadruple. The central structural object of the framework is a time indexed family of target submanifolds in the extended state space of parameters, velocities, and internal memory, which we call the Normally Attracting Invariant Manifold (NAIM) family. These submanifolds organize training dynamics into two timescales: a fast contraction of the velocity state onto the current target submanifold, followed by a slow descent along it. Using Riemannian backstepping, we construct a strict Lyapunov function on the extended state space and prove Uniform Ultimate Boundedness (UUB) of the optimization trajectory under a descent alignment assumption and bounded stochastic disturbances. Under the additional Polyak-Lojasiewicz (PL) condition, the Lyapunov function converges linearly to a bounded neighborhood of the optimum, defined as the drift of the target direction field between consecutive iterates after parallel transport. To validate the framework's generativity, we derive three concrete instantiations of the quadruple, each exercising a distinct design axis, and show on geometric diagnostics, image classification benchmarks, and language modeling that their empirical behavior matches the theoretical predictions. The framework casts optimizer design as controller synthesis, complementing heuristic perspectives by providing a constructive template with provable guarantees.

We introduce Continuous-Action World Models (CoAWM), a framework for learning cross-embodiment spatiotemporal dynamics priors that support robot action generation across embodiments and execution frequencies. At its core is a shared latent function space that captures the spatial and temporal structure of robot actions. We learn this space through two stages. First, function pretraining brings trajectories from different embodiments and sampling frequencies into a common representation, learning shared dynamics priors through action reconstruction and consistency across temporal sampling views. By accommodating heterogeneous robot trajectories, this stage broadens the range of data sources available for pretraining and facilitates scaling to larger and more diverse datasets. Second, we connect the pretrained function space to a world-model backbone, enabling observations, language instructions, and the current robot state to condition the prediction of a latent action function. A single decoder takes any query time and returns the corresponding end-effector action. By separating shared dynamics priors from task-conditioned action selection and temporal sampling, our approach provides a unified foundation for transferring robot experience across embodiments and control frequencies.