基于神经对偶评论家的一般效用强化学习样本高效方法
Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics
Ankur Naskar, Vaneet Aggarwal
AI 速览：作者提出一种自然演员-评论家方法，用神经评论家直接学习变分对偶伪奖励，以求解凹的一般效用强化学习（RLGU）问题，并给出样本复杂度分析。
Reinforcement learning with general utilities optimizes nonlinear functionals of policy-induced occupancy measures. We develop a natural actor-critic method for concave RLGU objectives using an exact variational dual representation, whose existence follows from concavity and differentiability, with explicit constructions for a broad class of separable occupancy utilities. Rather than estimating the occupancy measure and then forming the policy-gradient pseudo-reward, our method learns this pseudo-reward directly as a function of the state-action pair using a neural critic. The critic is trained by projected stochastic updates, while multilevel Monte Carlo combines outputs across training horizons to control its bias at logarithmic expected cost. Our analysis separates conditional actor bias from estimator second moments and accounts explicitly for dual optimization, critic approximation, and neural linearization errors. Under policy-transfer, dual-stability, and local critic-regularity conditions, the method achieves $\widetilde{O}(T^{-1/2})$ global convergence above policy, critic-class, and finite-width approximation errors using $\widetilde{O}(T)$ expected environment transitions and reference samples. For a fixed critic representation, this yields $\widetilde{O}(ε^{-2})$ sampling complexity above the approximation floor, covering infinite state-action spaces and occupancy utilities beyond fixed finite vectors of returns.