跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2509.26442 · v3

Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learning

Robbins-Siegmund 定理的扩展及其在强化学习中的应用

首次提交 2025-10-01 00:00
版本更新 2026-11-20 01:33
雷达观测 2026-11-20 12:52
arXiv ↗PDF ↗

摘要

The Robbins-Siegmund theorem establishes the convergence of stochastic processes that are almost supermartingales and is one of the most commonly used approaches for analyzing stochastic iterative algorithms in stochastic approximation and reinforcement learning (RL). However, its original form has a significant limitation as it requires the zero-order term to be summable. In many important RL applications, this summable condition, however, cannot be met. This limitation motivates us to extend the Robbins-Siegmund theorem for almost supermartingales where the zero-order term is not summable, but only square-summable. In particular, we introduce a novel and mild assumption on the increments of the stochastic processes. This together with the square-summable condition enables an almost sure convergence to a bounded set. Additionally, we further provide almost sure convergence rates, high probability concentration bounds, and LpL^p convergence rates. We then apply the new results to stochastic approximation and RL. Notably, we obtain the first almost sure convergence rate, the first high probability concentration bound, and the first LpL^p convergence rate for QQ-learning with linear function approximation.

AI 论文解读

基于摘要 · 非全文审阅

作者将 Robbins-Siegmund 定理中零阶项可和的条件放宽为平方可和,并引入对增量温和的新假设,从而建立几乎必然收敛到有界集及多种收敛速率,并应用于随机逼近与强化学习。

正在读取账号与研究记录…

版本差异

本站首次收录为 v3,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v3 · 2026-11-20 01:33 · 最新观测版本

Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learning

The Robbins-Siegmund theorem establishes the convergence of stochastic processes that are almost supermartingales and is one of the most commonly used approaches for analyzing stochastic iterative algorithms in stochastic approximation and reinforcement learning (RL). However, its original form has a significant limitation as it requires the zero-order term to be summable. In many important RL applications, this summable condition, however, cannot be met. This limitation motivates us to extend the Robbins-Siegmund theorem for almost supermartingales where the zero-order term is not summable, but only square-summable. In particular, we introduce a novel and mild assumption on the increments of the stochastic processes. This together with the square-summable condition enables an almost sure convergence to a bounded set. Additionally, we further provide almost sure convergence rates, high probability concentration bounds, and LpL^p convergence rates. We then apply the new results to stochastic approximation and RL. Notably, we obtain the first almost sure convergence rate, the first high probability concentration bound, and the first LpL^p convergence rate for QQ-learning with linear function approximation.

查看此版本解读 →查看此版本原文 ↗