跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.14988 · v2

Mixing Regularization for Average-Reward $Q$-Learning and $Q$-Planning

首次提交 2026-10-10 23:06
版本更新 2026-11-19 23:09
雷达观测 2026-11-20 14:52
arXiv ↗PDF ↗

摘要

The average-reward setting for reinforcement learning is challenging because, without discounting, the span-contraction of the Bellman operator depends on the mixing properties of the underlying Markov dynamics. Whereas existing work typically assumes contraction, for instance through environment mixing, we study regularization toward a given reward-agnostic, mm-step mixing policy to control the mixing properties of the learned policy. Our central mixing lemma quantifies how the reference policy's mixing properties transfer to the resulting learned policies. In particular, a single regularizer-dependent quantity simultaneously controls the contraction of the regularized Bellman operator and the exploration of the resulting policies. A second regularizer-dependent quantity controls the bias introduced by regularization, revealing a trade-off between convergence speed and approximation accuracy. Building on this result, we establish finite-time guarantees for regularized QQ-iteration, synchronous and single-trajectory QQ-learning, and QQ-planning. Under one-step mixing, our bounds for approximating the regularized fixed point match the classical model-query bound for relative value iteration and, up to logarithmic factors, an existing sample-complexity bound for synchronous QQ-learning. We also establish finite-time guarantees under multi-step mixing. For QQ-planning, we characterize how the planning horizon trades off boundary bias against sample complexity. Finally, we present numerical studies validating the analysis: (i) knowledge of a strong mixing policy improves convergence and exploration, with larger benefits under more severe bottlenecks; and (ii) the observed rankings of αα-divergence regularizers by bias and convergence speed are consistent with our theoretical analysis.

作者备注:10 pages main body, 58 pages references and appendix, 11 figures. Under review

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-19 23:09 · 最新观测版本

Mixing Regularization for Average-Reward $Q$-Learning and $Q$-Planning

The average-reward setting for reinforcement learning is challenging because, without discounting, the span-contraction of the Bellman operator depends on the mixing properties of the underlying Markov dynamics. Whereas existing work typically assumes contraction, for instance through environment mixing, we study regularization toward a given reward-agnostic, mm-step mixing policy to control the mixing properties of the learned policy. Our central mixing lemma quantifies how the reference policy's mixing properties transfer to the resulting learned policies. In particular, a single regularizer-dependent quantity simultaneously controls the contraction of the regularized Bellman operator and the exploration of the resulting policies. A second regularizer-dependent quantity controls the bias introduced by regularization, revealing a trade-off between convergence speed and approximation accuracy. Building on this result, we establish finite-time guarantees for regularized QQ-iteration, synchronous and single-trajectory QQ-learning, and QQ-planning. Under one-step mixing, our bounds for approximating the regularized fixed point match the classical model-query bound for relative value iteration and, up to logarithmic factors, an existing sample-complexity bound for synchronous QQ-learning. We also establish finite-time guarantees under multi-step mixing. For QQ-planning, we characterize how the planning horizon trades off boundary bias against sample complexity. Finally, we present numerical studies validating the analysis: (i) knowledge of a strong mixing policy improves convergence and exploration, with larger benefits under more severe bottlenecks; and (ii) the observed rankings of αα-divergence regularizers by bias and convergence speed are consistent with our theoretical analysis.

查看此版本解读 →查看此版本原文 ↗