Delayed homomorphic reinforcement learning for environments with delayed feedback
Jongsoo Lee, Jangwon Kim, Soohee Han
原文摘要
Reinforcement learning in real-world systems often involves delayed feedback, which violates the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches address this issue by augmenting the state with action histories, but suffer from state-space explosion and the resulting sample-complexity burden. Despite recent progress, state-of-the-art augmentation-based methods either primarily alleviate this burden for the critic or employ separate mechanisms for the actor and critic. To address this limitation, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that introduces a belief-equivalence relation over the augmented state space to collapse control-redundant augmented states into a compact abstract state space. This framework yields exact abstraction under deterministic dynamics and approximate abstraction under stochastic dynamics, providing a unified abstraction mechanism that benefits both the actor and critic. In finite domains, exact abstraction preserves optimality and recovers the delay-free sample-complexity order, while approximate abstraction admits a bound on the value loss of the resulting policy. For continuous domains, we introduce deep delayed homomorphic policy gradient (D$^2$HPG), a deep actor-critic instantiation of the DHRL framework. Experiments on MuJoCo continuous-control benchmarks demonstrate that D$^2$HPG outperforms strong augmentation-based baselines, especially in long-delay regimes. Code is available at https://github.com/JongsooLee/DHRL.
cs.LG · cs.AI