在智能体强化学习中稳定化同策略定向引导
Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning
Kevin Frans, Alexander M Rush, Sergey Levine, Pieter Abbeel, Ashvin Nair
AI 速览：作者提出并分析“定向引导”框架，通过同策略自蒸馏从逐轮文本反馈中学习，并在智能体多轮软件工程环境中探索如何稳定训练且不损害基线RL性能。
Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize such structures to provide a rich directional signal that enables improving on otherwise difficult to optimize rewards? In this work, we examine a framework we refer to as targeted guidance that learns from per-turn textual feedback via an on-policy self-distillation objective. Our focus is not only on directly improving hard-to-optimize behaviors, but on how to do so throughout the training process without degrading baseline RL performance. Over a range of settings across an agentic multi-turn software engineering environment, we find success by combining targeted self-distillation with a per-criteria RL reward, and by limiting feedback to turns where the policy specifically fails. We further examine the design choices that lead to stable entropy and reduce oscillatory behavior throughout training.