← 研踪 PaperTrailarXiv:2605.19235 · v2
GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learning
首次提交 2026-05-19 09:07
版本更新 2026-11-09 12:58
雷达观测 2026-11-10 13:18
摘要
Competitive multi-agent reinforcement learning in imperfect-information games requires agents to act under partial observability and against adversarial opponents, necessitating stochastic policies. While self-play reinforcement learning with Proximal Policy Optimization (PPO) has achieved strong empirical success, its standard advantage estimator, generalized advantage estimation, suffers from additional variance due to the sampling of stochastic future actions. This variance is amplified in equilibrium self-play because of the stochastic nature of the equilibrium policy and persists even when the critic is exact. We address this bottleneck by introducing $Q$-boosting, a variance-reduced advantage estimator based on a centralized action-value critic, and propose Variance-Reduced Policy Optimization (VRPO), incorporating this new estimator. The algorithm replaces sampled multi-step backups with a multi-step Expected SARSA$(λ)$ trace, computing policy expectations at each step to average out action-sampling noise, while retaining PPO's clipped objective and on-policy actor updates. Empirically, VRPO achieves strong performance in various games: In mid-sized games where exploitability is computable, VRPO lowers the exact exploitability by more than 15% compared to tuned baselines. In Dou Dizhu, it sets a new state of the art over PerfectDou. In Heads-Up No-Limit Texas Hold'em, it posts a positive win rate against Slumbot without any blueprint raise sizes, search, or subgame solving at test time.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-09 12:58 · 最新观测版本
GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learning
Competitive multi-agent reinforcement learning in imperfect-information games requires agents to act under partial observability and against adversarial opponents, necessitating stochastic policies. While self-play reinforcement learning with Proximal Policy Optimization (PPO) has achieved strong empirical success, its standard advantage estimator, generalized advantage estimation, suffers from additional variance due to the sampling of stochastic future actions. This variance is amplified in equilibrium self-play because of the stochastic nature of the equilibrium policy and persists even when the critic is exact. We address this bottleneck by introducing $Q$-boosting, a variance-reduced advantage estimator based on a centralized action-value critic, and propose Variance-Reduced Policy Optimization (VRPO), incorporating this new estimator. The algorithm replaces sampled multi-step backups with a multi-step Expected SARSA$(λ)$ trace, computing policy expectations at each step to average out action-sampling noise, while retaining PPO's clipped objective and on-policy actor updates. Empirically, VRPO achieves strong performance in various games: In mid-sized games where exploitability is computable, VRPO lowers the exact exploitability by more than 15% compared to tuned baselines. In Dou Dizhu, it sets a new state of the art over PerfectDou. In Heads-Up No-Limit Texas Hold'em, it posts a positive win rate against Slumbot without any blueprint raise sizes, search, or subgame solving at test time.
查看此版本解读 →查看此版本原文 ↗