← 研踪 PaperTrailarXiv:2605.19919 · v2
Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning
首次提交 2026-05-19 22:43
版本更新 2026-11-07 22:39
雷达观测 2026-11-10 11:14
摘要
Pretrained imitation policies have become a strong foundation for robot manipulation, but online adaptation can further improve policy performance within their pretrained behavioral support. A central question is therefore how reinforcement learning (RL) should adapt policies after offline pretraining. Existing lightweight methods commonly apply residual corrections directly in action space. We investigate how the steering interface affects online RL. In this work, we propose Z-Perturbation Reinforcement Learning (ZPRL), an approach that steers pretrained policies through a compact bottleneck latent rather than through policy weights or output actions. Building on an offline-learned bottleneck policy interface introduced by previous work, ZPRL turns this representation into an online RL control space through reward-driven residual perturbations while freezing the pretrained policy. We instantiate ZPRL on flow-matching policies and evaluate it on eight simulation tasks and four real-world tasks. Across diverse manipulation settings, ZPRL improves sample efficiency with competitive or better final performance. In the real world, ZPRL improves the average success rate on four tasks by 33.7 percentage points over imitation base policies. Our qualitative and quantitative results also indicate that ZPRL combines task success with smooth actions without explicit temporal filtering or smoothness regularization. These results suggest that a compact, task-aligned bottleneck latent provides an effective interface for online RL adaptation. Videos and code can be found at https://manutdmoon.github.io/ZPRL/.
作者备注：Accepted by IEEE T-RL. Experiments and description updated
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 22:39 · 最新观测版本
Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning
Pretrained imitation policies have become a strong foundation for robot manipulation, but online adaptation can further improve policy performance within their pretrained behavioral support. A central question is therefore how reinforcement learning (RL) should adapt policies after offline pretraining. Existing lightweight methods commonly apply residual corrections directly in action space. We investigate how the steering interface affects online RL. In this work, we propose Z-Perturbation Reinforcement Learning (ZPRL), an approach that steers pretrained policies through a compact bottleneck latent rather than through policy weights or output actions. Building on an offline-learned bottleneck policy interface introduced by previous work, ZPRL turns this representation into an online RL control space through reward-driven residual perturbations while freezing the pretrained policy. We instantiate ZPRL on flow-matching policies and evaluate it on eight simulation tasks and four real-world tasks. Across diverse manipulation settings, ZPRL improves sample efficiency with competitive or better final performance. In the real world, ZPRL improves the average success rate on four tasks by 33.7 percentage points over imitation base policies. Our qualitative and quantitative results also indicate that ZPRL combines task success with smooth actions without explicit temporal filtering or smoothness regularization. These results suggest that a compact, task-aligned bottleneck latent provides an effective interface for online RL adaptation. Videos and code can be found at https://manutdmoon.github.io/ZPRL/.
查看此版本解读 →查看此版本原文 ↗