跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2603.28595 · v3

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes

首次提交 2026-03-30 23:41
版本更新 2026-11-18 18:42
雷达观测 2026-11-19 15:04
arXiv ↗PDF ↗

摘要

Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and define "implicit" policies through the accumulated critic estimates, rather than learning an explicitly parameterized actor. Such policies can be computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable logit-matching regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that, when the policy features are sufficiently expressive, the resulting algorithm achieves O~(ε−4)\widetilde{O}(ε^{-4}) and O~(ε−2)\widetilde{O}(ε^{-2}) sample complexity in the on-policy and off-policy settings, respectively. Our results match the state-of-the-art dependence on εε in prior theoretical work, while our algorithm is more aligned with practice.

作者备注:AISTATS 2026

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v3,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v3 · 2026-11-18 18:42 · 最新观测版本

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes

Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and define "implicit" policies through the accumulated critic estimates, rather than learning an explicitly parameterized actor. Such policies can be computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable logit-matching regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that, when the policy features are sufficiently expressive, the resulting algorithm achieves O~(ε−4)\widetilde{O}(ε^{-4}) and O~(ε−2)\widetilde{O}(ε^{-2}) sample complexity in the on-policy and off-policy settings, respectively. Our results match the state-of-the-art dependence on εε in prior theoretical work, while our algorithm is more aligned with practice.

查看此版本解读 →查看此版本原文 ↗