← 研踪 PaperTrailarXiv:2604.03641 · v3
Delayed homomorphic reinforcement learning for environments with delayed feedback
首次提交 2026-04-04 16:38
版本更新 2026-11-11 20:51
雷达观测 2026-11-13 14:16
摘要
Reinforcement learning in real-world systems often involves delayed feedback, which violates the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches address this issue by augmenting the state with action histories, but suffer from state-space explosion and the resulting sample-complexity burden. Despite recent progress, state-of-the-art augmentation-based methods either primarily alleviate this burden for the critic or employ separate mechanisms for the actor and critic. To address this limitation, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that introduces a belief-equivalence relation over the augmented state space to collapse control-redundant augmented states into a compact abstract state space. This framework yields exact abstraction under deterministic dynamics and approximate abstraction under stochastic dynamics, providing a unified abstraction mechanism that benefits both the actor and critic. In finite domains, exact abstraction preserves optimality and recovers the delay-free sample-complexity order, while approximate abstraction admits a bound on the value loss of the resulting policy. For continuous domains, we introduce deep delayed homomorphic policy gradient (DHPG), a deep actor-critic instantiation of the DHRL framework. Experiments on MuJoCo continuous-control benchmarks demonstrate that DHPG outperforms strong augmentation-based baselines, especially in long-delay regimes. Code is available at https://github.com/JongsooLee/DHRL.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-11 20:51 · 最新观测版本
Delayed homomorphic reinforcement learning for environments with delayed feedback
Reinforcement learning in real-world systems often involves delayed feedback, which violates the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches address this issue by augmenting the state with action histories, but suffer from state-space explosion and the resulting sample-complexity burden. Despite recent progress, state-of-the-art augmentation-based methods either primarily alleviate this burden for the critic or employ separate mechanisms for the actor and critic. To address this limitation, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that introduces a belief-equivalence relation over the augmented state space to collapse control-redundant augmented states into a compact abstract state space. This framework yields exact abstraction under deterministic dynamics and approximate abstraction under stochastic dynamics, providing a unified abstraction mechanism that benefits both the actor and critic. In finite domains, exact abstraction preserves optimality and recovers the delay-free sample-complexity order, while approximate abstraction admits a bound on the value loss of the resulting policy. For continuous domains, we introduce deep delayed homomorphic policy gradient (DHPG), a deep actor-critic instantiation of the DHRL framework. Experiments on MuJoCo continuous-control benchmarks demonstrate that DHPG outperforms strong augmentation-based baselines, especially in long-delay regimes. Code is available at https://github.com/JongsooLee/DHRL.
查看此版本解读 →查看此版本原文 ↗