跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2610.02527 · v2

CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization

首次提交 2026-10-02 05:57
版本更新 2026-11-18 08:11
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

Learned visual reward models score a robot's execution from camera images and an instruction, and are increasingly used to fine-tune robot policies. Two checks decide whether to trust one: the reward model separates successes from failures, and task success rises during training. We show that both checks can pass while the policy learns to act on the wrong object. When a diffusion policy is fine-tuned against the reward model Robometer to open a drawer, task success rises by 10.2 percentage points. Wrong-object failures, which open a different drawer, rise by 10.9 points, whereas the same training on the simulator's task-completion signal raises success alone. A second reward model, RoboReward-8B, behaves the same way, and the effect reappears on a physical robot. This happens because of how the reward scores the policy's own failures. Most failures make little progress and score low, while a wrong-object failure looks like progress and scores well above them. Optimization shifts a policy toward outcomes that score above the executions they compete with, the executions they can take the place of. It therefore turns low-scoring failures into both successes and wrong-object failures, even when the reward model scores successes higher. This rule needs only executions of the initial policy, so it can be tested before training. On a stacking task we predicted which of three object placements Robometer would exploit, and training raised wrong-object stacks from 20% to 69% in that placement and in neither of the others. The same rule gives a check to run before optimization and a repair that needs no labels: scoring each execution again on frames re-rendered with the objects the instruction does not name put back removes the amplification on both tasks and on the robot.

作者备注:36 pages

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-18 08:11 · 最新观测版本

CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization

Learned visual reward models score a robot's execution from camera images and an instruction, and are increasingly used to fine-tune robot policies. Two checks decide whether to trust one: the reward model separates successes from failures, and task success rises during training. We show that both checks can pass while the policy learns to act on the wrong object. When a diffusion policy is fine-tuned against the reward model Robometer to open a drawer, task success rises by 10.2 percentage points. Wrong-object failures, which open a different drawer, rise by 10.9 points, whereas the same training on the simulator's task-completion signal raises success alone. A second reward model, RoboReward-8B, behaves the same way, and the effect reappears on a physical robot. This happens because of how the reward scores the policy's own failures. Most failures make little progress and score low, while a wrong-object failure looks like progress and scores well above them. Optimization shifts a policy toward outcomes that score above the executions they compete with, the executions they can take the place of. It therefore turns low-scoring failures into both successes and wrong-object failures, even when the reward model scores successes higher. This rule needs only executions of the initial policy, so it can be tested before training. On a stacking task we predicted which of three object placements Robometer would exploit, and training raised wrong-object stacks from 20% to 69% in that placement and in neither of the others. The same rule gives a check to run before optimization and a repair that needs no labels: scoring each execution again on frames re-rendered with the objects the instruction does not name put back removes the amplification on both tasks and on the robot.

查看此版本解读 →查看此版本原文 ↗