Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics
Ankur Naskar, Vaneet Aggarwal
AI 速览：作者提出一种自然演员-评论家方法，用神经评论家直接学习变分对偶伪奖励，以求解凹的一般效用强化学习（RLGU）问题，并给出样本复杂度分析。
Reinforcement learning with general utilities optimizes nonlinear functionals of policy-induced occupancy measures. We develop a natural actor-critic method for concave RLGU objectives using an exact variational dual representation, whose existence follows from concavity and differentiability, with explicit constructions for a broad class of separable occupancy utilities. Rather than estimating the occupancy measure and then forming the policy-gradient pseudo-reward, our method learns this pseudo-reward directly as a function of the state-action pair using a neural critic. The critic is trained by projected stochastic updates, while multilevel Monte Carlo combines outputs across training horizons to control its bias at logarithmic expected cost. Our analysis separates conditional actor bias from estimator second moments and accounts explicitly for dual optimization, critic approximation, and neural linearization errors. Under policy-transfer, dual-stability, and local critic-regularity conditions, the method achieves $\widetilde{O}(T^{-1/2})$ global convergence above policy, critic-class, and finite-width approximation errors using $\widetilde{O}(T)$ expected environment transitions and reference samples. For a fixed critic representation, this yields $\widetilde{O}(ε^{-2})$ sampling complexity above the approximation floor, covering infinite state-action spaces and occupancy utilities beyond fixed finite vectors of returns.
We obtain the approximation coefficient $α_0=54440750000/117486474169>0.463378$ for maximizing nonnegative, non-monotone DR-submodular functions over compact convex down-closed subsets of the unit cube, with queries allowed throughout the cube. This improves the previous best coefficient, $4/9$. A coordinatewise map, exponential on small coordinates and linear-fractional on large ones, turns the learner's state, a feasible point, into a feasible action whose value is at least $α_0$ times that of any feasible point, up to a linear term. The proof is analytical and reduces to one-dimensional inequalities. The linear term has an unbiased estimator from one noisy-gradient query, so online maximization reduces to linear learning. Under a uniform bound on the gradients, the expected approximate regret is $O(\sqrt T)$ with projected ascent and $O(T^{3/4})$ with one linear-maximization call per round; with one noisy value per round, it is $O(T^{3/4})$ and $O(T^{7/8})$. Online-to-batch conversion gives offline guarantees. With exact values and gradients, contracting the objective toward a feasible interior point yields an expected $(α_0-\varepsilon)$-approximation for every nonnegative $C^1$ DR-submodular objective, with oracle counts polynomial in $n$ and $\varepsilon^{-1}$ and no gradient or smoothness parameter. Sampling and standard rounding give $(α_0-\varepsilon)$-approximation for submodular set functions under one matroid or a fixed number of knapsack constraints.