跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2607.11505 · v3

Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Update for Large Language Models

首次提交 2026-07-13 20:56
版本更新 2026-11-16 20:58
雷达观测 2026-11-17 17:11
arXiv ↗PDF ↗

摘要

Reward-based reinforcement learning (RL) improves large language models but incurs substantial costs when repeated across targets. On-policy distillation (OPD) reuses experts by matching their distributions on target-generated trajectories, yet differing base policies can confound reward-induced updates with pre-existing policy differences, potentially distorting transferred update directions. We propose Proxy OPD (P-OPD), a post-training method that transfers reward-induced relative policy updates using the target's frozen base policy as a reference. A proxy undergoes reward-guided exploration, yielding token-level log-probability changes between its initial and optimized policies. These signals guide learning on target-generated trajectories, with calibration controlling signal absorption, enabling reuse of reward-induced improvements across targets. Experiments on Qwen3-family targets in mathematical reasoning, code generation, and instruction following show that P-OPD improves targets and outperforms conventional OPD in most settings. For 1.7B-to-8B transfer, average Math Mean@16 and Code Mean@8 scores exceed OPD by 8.3 and 3.4 points, respectively. Mechanism analyses show more faithful reward-induced supervision and controllable adaptation. Cross-scale and cross-generation experiments confirm signal transferability and reusability, while multi-domain distillation reveals synergistic gains and cross-domain interference. In the measured 4B-to-8B setting, target adaptation after proxy exploration takes approximately 15\% of direct target RL time; repeated signal reuse amortizes the one-time exploration cost across compatible models. These findings support relative policy updates as effective, transferable, and composable supervision for an ``explore once, transfer multiple times'' post-training paradigm.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v3,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v3 · 2026-11-16 20:58 · 最新观测版本

Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Update for Large Language Models

Reward-based reinforcement learning (RL) improves large language models but incurs substantial costs when repeated across targets. On-policy distillation (OPD) reuses experts by matching their distributions on target-generated trajectories, yet differing base policies can confound reward-induced updates with pre-existing policy differences, potentially distorting transferred update directions. We propose Proxy OPD (P-OPD), a post-training method that transfers reward-induced relative policy updates using the target's frozen base policy as a reference. A proxy undergoes reward-guided exploration, yielding token-level log-probability changes between its initial and optimized policies. These signals guide learning on target-generated trajectories, with calibration controlling signal absorption, enabling reuse of reward-induced improvements across targets. Experiments on Qwen3-family targets in mathematical reasoning, code generation, and instruction following show that P-OPD improves targets and outperforms conventional OPD in most settings. For 1.7B-to-8B transfer, average Math Mean@16 and Code Mean@8 scores exceed OPD by 8.3 and 3.4 points, respectively. Mechanism analyses show more faithful reward-induced supervision and controllable adaptation. Cross-scale and cross-generation experiments confirm signal transferability and reusability, while multi-domain distillation reveals synergistic gains and cross-domain interference. In the measured 4B-to-8B setting, target adaptation after proxy exploration takes approximately 15\% of direct target RL time; repeated signal reuse amortizes the one-time exploration cost across compatible models. These findings support relative policy updates as effective, transferable, and composable supervision for an ``explore once, transfer multiple times'' post-training paradigm.

查看此版本解读 →查看此版本原文 ↗