跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2411.13711 · v2

Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise

马尔可夫噪声下随机逼近与强化学习的几乎必然收敛速率与集中不等式

首次提交 2024-11-21 05:09
版本更新 2026-11-20 02:10
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

This paper establishes, to our knowledge, the first almost sure convergence rate and the first maximal concentration bound with sub-Weibull tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in LpL^p. Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. Applications yield the first almost sure convergence rate for QQ-learning without count-based learning rates and the first maximal concentration bound for off-policy temporal difference learning, both with Markovian samples.

AI 论文解读

基于摘要 · 非全文审阅

该论文为马尔可夫噪声下的收缩随机逼近算法建立了首个几乎必然收敛速率和首个具有次Weibull尾部的最大集中不等式,并应用于Q学习和离策略时序差分学习。

正在读取账号与研究记录…

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-20 02:10 · 最新观测版本

Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise

This paper establishes, to our knowledge, the first almost sure convergence rate and the first maximal concentration bound with sub-Weibull tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in LpL^p. Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. Applications yield the first almost sure convergence rate for QQ-learning without count-based learning rates and the first maximal concentration bound for off-policy temporal difference learning, both with Markovian samples.

查看此版本解读 →查看此版本原文 ↗