Reframe:面向智能体视觉推理的观测对齐多教师在线策略蒸馏
Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning
Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
AI 速读 基于摘要
提出 Reframe 后训练框架,把工具返回的观测状态而非仅输出分布作为对齐对象,结合 GD-GRPO 与 OG-MOPD 两阶段提升视觉智能体的工具使用与蒸馏效果。
cs.CV
查看原文摘要
Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.