奖励膨胀:强化学习的一种健康刺激
Reward Inflation: A Healthy Stimulus for Reinforcement Learning
Ganghun Lee, Minji Kim, Minsu Lee, Byoung-Tak Zhang
AI 速读 基于摘要
作者提出在训练过程中逐步放大奖励的“奖励膨胀”方法,理论分析其能隐式加权近期经验并抑制休眠神经元,实验表明适度膨胀有益于多类任务,并给出自适应变体Fed。
cs.LG
查看原文摘要
Reward serves as the primary learning signal in reinforcement learning (RL). However, while reward magnitudes are typically held fixed throughout training, their temporal modulation remains underexplored. In this paper, we propose reward inflation, a gradual scaling of rewards over the course of training, and show that it can act as a healthy stimulus for RL. Theoretically, reward inflation induces an implicit recency weighting that upweights recent transitions during policy updates, enabling faster adaptation. We further show that reward inflation suppresses the emergence of dormant neurons by sustaining gradient signals as the policy saturates, suggesting a possible contribution to plasticity preservation. Empirical results on ALE games and MuJoCo tasks corroborate these findings, showing that an appropriate level of reward inflation benefits a broad range of tasks. Finally, we introduce Fed, an adaptive variant that adjusts the inflation level on the fly, and find that it often improves upon fixed inflation. Source code is available at https://github.com/northadventure/RewardInflation.