LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
Yujin Kim, Namgyu Ho, Sangmin Hwang, Joonkee Kim, Yongjin Yang, Sangmin Bae, Seungone Kim, Jaehun Jung 等 10 位作者
原文摘要
Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design preserves the original prompt while adding constraints to increase difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, on average and on four of five metrics, suggesting prompt adaptation as an underexplored axis of policy-awareness in non-verifiable RL for constraint-heavy instruction following.
cs.AI