跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.21879 · v1

Adaptive Memory for Online Reinforcement Learning under Sparse Model Changes

首次提交 2026-11-17 04:44
版本更新 2026-11-17 04:44
雷达观测 2026-11-18 19:44
arXiv ↗PDF ↗

摘要

We study the discounted reinforcement learning (RL) problem in a setting where the underlying environment changes between deployments. Each deployment provides HH observations from a fixed MDP. The reward function and transition kernel are fixed within each deployment. Across deployments, at most an ε>0ε>0 fraction of the reward and transition kernel entries may change. In this setting, learning each model to deploy from scratch may discard useful observations from past deployments, whereas pooling all past observations can bias the estimates of entries that have already changed. We then propose an algorithm that for each state and action pair, selects how much history to reuse to estimate the reward and probabilities of the next state. We start with observations from the current deployment and add earlier deployment observations one at a time, the most recent first. After each addition, we check whether a single average reward agrees with the estimates from all histories considered until that point. We apply the same check to the distribution of next states. For each estimate, we leverage the last history that passes these aforementioned checks. The optimal control policy is then computed on top of the fitted MDP. We characterize the average gap between the optimal return and the return of the learner's policies over a finite number of deployments. We demonstrate that when 0<∣S∣ε≤20< |S|ε\leq 2, with ∣S∣|S| being the cardinality of the state space, our algorithm needs only the observations from the most recent O(1/ε)O(1/ε) deployments, and achieves a long-run average gap of at most O~(ε/H)\widetilde{O}(\sqrt{ε/H}).

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-17 04:44 · 最新观测版本

Adaptive Memory for Online Reinforcement Learning under Sparse Model Changes

We study the discounted reinforcement learning (RL) problem in a setting where the underlying environment changes between deployments. Each deployment provides HH observations from a fixed MDP. The reward function and transition kernel are fixed within each deployment. Across deployments, at most an ε>0ε>0 fraction of the reward and transition kernel entries may change. In this setting, learning each model to deploy from scratch may discard useful observations from past deployments, whereas pooling all past observations can bias the estimates of entries that have already changed. We then propose an algorithm that for each state and action pair, selects how much history to reuse to estimate the reward and probabilities of the next state. We start with observations from the current deployment and add earlier deployment observations one at a time, the most recent first. After each addition, we check whether a single average reward agrees with the estimates from all histories considered until that point. We apply the same check to the distribution of next states. For each estimate, we leverage the last history that passes these aforementioned checks. The optimal control policy is then computed on top of the fitted MDP. We characterize the average gap between the optimal return and the return of the learner's policies over a finite number of deployments. We demonstrate that when 0<∣S∣ε≤20< |S|ε\leq 2, with ∣S∣|S| being the cardinality of the state space, our algorithm needs only the observations from the most recent O(1/ε)O(1/ε) deployments, and achieves a long-run average gap of at most O~(ε/H)\widetilde{O}(\sqrt{ε/H}).

查看此版本解读 →查看此版本原文 ↗