Sample-efficient reinforcement learning typically uses a fixed reward function both to guide exploration and to supervise learning. However, the same experience can provide supervision under multiple related reward specifications, even when the objective of interest remains fixed. We introduce Reward-Conditioned Reinforcement Learning (RCRL), an off-policy approach that collects experience under a nominal objective while training a reward-conditioned agent using counterfactual rewards recomputed from replay. RCRL constructs alternative reward specifications by perturbing the nominal reward, providing diverse supervision without additional environment interaction. Across single-task, multi-task, and vision-based control benchmarks, RCRL improves nominal task performance and sample efficiency when integrated with strong RL backbones. As a secondary capability, the learned policies support zero-shot behavioral adjustment toward related objectives without further training. These results demonstrate that diverse reward supervision can improve learning of a predefined objective while also enabling policy steerability.
Kevin Qiu, Kyle Liam Walker, Mike Yan Michelis, Marek Cygan, Josie Hughes
原文摘要
Calibrating a soft aquatic robot simulator requires matching coupled deformation and propulsion from limited pool recordings. Swim2Real uses a vision-language model (VLM) to compare real and simulated swimming videos and propose updates to 16 simulator parameters. A backtracking line search tests each direction and accepts updates that reduce marker error. To test whether the proposals carry useful directions, we compare them with random and signed-permuted directions under the same line search and simulator budget. In an independent, preregistered fifty-seed replication, Swim2Real reduces the mean best-so-far error over 40 simulator evaluations by 63.4 mm against random directions and 53.2 mm against permuted directions. Both paired comparisons pass Holm correction ($p<10^{-5}$), conditional on run completion. A 24-seed experiment with a second VLM supports the same direction effect. Backtracking averages 16.1 VLM proposals per run versus 39 for Vid2Sid, which attains lower error at the same simulator budget. Ten hardware trials demonstrate open-loop execution of simulator-generated motor commands. Project page and videos: https://swim2real.github.io.