← 研踪 PaperTrailarXiv:2605.28751 · v2
Correctness vs Efficiency: Behavior Control with Weight-Space Extrapolation in Code RL
首次提交 2026-05-28 01:09
版本更新 2026-11-09 23:23
雷达观测 2026-11-10 15:22
摘要
A model trained with an imperfect reward in RL may learn to exploit the reward with undesired behavior. We study a controlled form of this problem in competitive programming. A program may pass small tests while violating the intended efficiency requirement on larger inputs. We construct a family of rewards that progressively relax the efficiency constraint, making the reward increasingly easy to exploit, and train separate policies. This sweep reveals a correctness-efficiency frontier. Relaxing the reward leads to less efficient programs and more resource-limit failures. Stricter rewards suppress these failures but increase correctness failures near the model's capability boundary. This frontier is also reflected in weight space. Interpolating between the resulting models reproduces the same behavioral change. More surprisingly, extrapolation in both directions further amplifies or suppresses it beyond the trained checkpoints. Extensive experiments show that this pattern holds across pure reasoning, tool use, and agentic coding, and at both 32B and 7B scales. This suggests a simple intervention for this form of reward hacking: use a loose reward to amplify the undesired behavior, use the weight-space difference to a strict-reward model as its steering vector, and then extrapolate past the strict model to suppress it.
作者备注：Accepted at NeurIPS 2026. Camera-ready version, 54 pages
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-09 23:23 · 最新观测版本
Correctness vs Efficiency: Behavior Control with Weight-Space Extrapolation in Code RL
A model trained with an imperfect reward in RL may learn to exploit the reward with undesired behavior. We study a controlled form of this problem in competitive programming. A program may pass small tests while violating the intended efficiency requirement on larger inputs. We construct a family of rewards that progressively relax the efficiency constraint, making the reward increasingly easy to exploit, and train separate policies. This sweep reveals a correctness-efficiency frontier. Relaxing the reward leads to less efficient programs and more resource-limit failures. Stricter rewards suppress these failures but increase correctness failures near the model's capability boundary. This frontier is also reflected in weight space. Interpolating between the resulting models reproduces the same behavioral change. More surprisingly, extrapolation in both directions further amplifies or suppresses it beyond the trained checkpoints. Extensive experiments show that this pattern holds across pure reasoning, tool use, and agentic coding, and at both 32B and 7B scales. This suggests a simple intervention for this form of reward hacking: use a loose reward to amplify the undesired behavior, use the weight-space difference to a strict-reward model as its steering vector, and then extrapolate past the strict model to suppress it.
查看此版本解读 →查看此版本原文 ↗