← 研踪 PaperTrailarXiv:2607.04412 · v3
LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
首次提交 2026-07-06 01:05
版本更新 2026-11-16 19:16
雷达观测 2026-11-17 17:11
摘要
Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design preserves the original prompt while adding constraints to increase difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, on average and on four of five metrics, suggesting prompt adaptation as an underexplored axis of policy-awareness in non-verifiable RL for constraint-heavy instruction following.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-16 19:16 · 最新观测版本
LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL
Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design preserves the original prompt while adding constraints to increase difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, on average and on four of five metrics, suggesting prompt adaptation as an underexplored axis of policy-awareness in non-verifiable RL for constraint-heavy instruction following.
查看此版本解读 →查看此版本原文 ↗