← 研踪 PaperTrailarXiv:2411.13711 · v2
Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise
马尔可夫噪声下随机逼近与强化学习的几乎必然收敛速率与集中不等式
首次提交 2024-11-21 05:09
版本更新 2026-11-20 02:10
雷达观测 2026-11-20 12:52
摘要
This paper establishes, to our knowledge, the first almost sure convergence rate and the first maximal concentration bound with sub-Weibull tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in . Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. Applications yield the first almost sure convergence rate for -learning without count-based learning rates and the first maximal concentration bound for off-policy temporal difference learning, both with Markovian samples.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-20 02:10 · 最新观测版本
Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise
This paper establishes, to our knowledge, the first almost sure convergence rate and the first maximal concentration bound with sub-Weibull tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in . Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. Applications yield the first almost sure convergence rate for -learning without count-based learning rates and the first maximal concentration bound for off-policy temporal difference learning, both with Markovian samples.
查看此版本解读 →查看此版本原文 ↗