Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
Yuheng Zhang, Wenlin Yao, Changlong Yu, Yao Liu, Qingyu Yin, Bing Yin, Hyokun Yun, Lihong Li
Group relative policy optimization (GRPO) trains large language models with verifiable rewards by normalizing rewards within response groups. When every response in a group receives the same reward, the advantages vanish and the group provides no policy gradient signal. We propose Adaptive Rollout and Response Reuse Policy Optimization (AR3PO) to obtain informative groups with fewer generations. Adaptive rollout samples responses to each prompt in stages and stops sampling once the model generates a correct response. If all sampled responses are incorrect, response reuse replaces one of them with a past correct response to the same prompt, when available, and recomputes the group advantages. Policy-recomputed reuse trains on the updated groups using probabilities recomputed under the rollout policy; advantage-only reuse uses historical responses only to compute advantages. Across four mathematical benchmarks with 7B and 8B models, AR3PO consistently improves over GRPO and achieves performance comparable to DAPO while requiring up to $4.2\times$ fewer generated responses. Experiments with a 32B model also show comparable performance to DAPO at a similar training stage with substantially fewer generated responses.
尚无 AI 解读，展示原文摘要