奖励塑形以缓解RLHF中的奖励黑客问题
Reward Shaping to Mitigate Reward Hacking in RLHF
Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, Yanghua Xiao, Yuxia Wang
AI 速读 基于摘要
提出Preference as Reward（PAR）奖励塑形方法，将中心化奖励经sigmoid变换，以有界且快速上升后饱和的信号缓解RLHF中的奖励黑客并稳定训练。
cs.LG · cs.AI · cs.CL
查看原文摘要
Reinforcement learning from human feedback (RLHF) is widely used to align large language models with human preferences, but it remains vulnerable to \emph{reward hacking}, whereby a policy exploits flaws in the reward function at the expense of the intended behavior. Reward shaping can stabilize training and partially mitigate reward hacking, yet its design principles remain underexplored. Our systematic study of common shaping methods suggests that shaped rewards should be bounded and should rise rapidly before gradually saturating. We introduce \emph{Preference as Reward} (PAR), which applies a sigmoid to centered rewards to produce a signal that reflects the reward model's latent preferences. We show that PAR has a variance-reduction property that stabilizes training and substantially widens the window for effective early stopping. Using Proximal Policy Optimization (PPO) and Gemma2-2B, PAR consistently outperforms the other shaping methods evaluated and demonstrates high data efficiency and robustness. Comparisons with unshaped rewards across four base models and four reinforcement learning algorithms show that PAR is effective across models and particularly beneficial for actor--critic methods when value estimates become unstable. The code is available at https://github.com/PorUna-byte/PAR.