← 研踪 PaperTrailarXiv:2609.38850 · v2
Beyond Outcome Rewards: Post-Rationale Distribution Optimization for Selective Prediction in Reasoning Models
首次提交 2026-09-30 11:08
版本更新 2026-11-02 03:28
雷达观测 2026-11-04 23:43
摘要
Reasoning models are trained to produce correct answers, yet selective deployment requires ranking which answers to accept or defer, and outcome-reward training does not directly optimize the post-rationale distribution used for that decision. We identify two mismatches: expected correctness is mode-seeking rather than distribution-eliciting, and scoring an average across rationales rewards disagreement unavailable to a single-rationale decision. We introduce Post-Rationale Distribution Optimization (PRDO), which applies a proper score to the answer distribution after each sampled rationale, without target rationales or separate confidence labels. Across four 0.6--3.8-billion-parameter models on MMLU-Pro, PRDO improves Brier score, error ranking and secondary accuracy relative to budget-matched, temperature-scaled GRPO. At 20\% realized error, oracle coverage rises by \npj{Q3a_vs_grpo.qwen3-0.6b.mp.oracle_cov@0.2.ppabs}--\npj{Q3a_vs_grpo.phi-4-mini.mp.oracle_cov@0.2.ppabs} percentage points, and ranking gains transfer to unseen tasks. Matched controls identify proper, per-rationale scoring as the active design choices, while verification tasks show the boundary: better selective ranking can coexist with lower full-coverage accuracy. Code is available at https://github.com/ZimmyGao/openjev-rlcd
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-02 03:28 · 最新观测版本
Beyond Outcome Rewards: Post-Rationale Distribution Optimization for Selective Prediction in Reasoning Models
Reasoning models are trained to produce correct answers, yet selective deployment requires ranking which answers to accept or defer, and outcome-reward training does not directly optimize the post-rationale distribution used for that decision. We identify two mismatches: expected correctness is mode-seeking rather than distribution-eliciting, and scoring an average across rationales rewards disagreement unavailable to a single-rationale decision. We introduce Post-Rationale Distribution Optimization (PRDO), which applies a proper score to the answer distribution after each sampled rationale, without target rationales or separate confidence labels. Across four 0.6--3.8-billion-parameter models on MMLU-Pro, PRDO improves Brier score, error ranking and secondary accuracy relative to budget-matched, temperature-scaled GRPO. At 20\% realized error, oracle coverage rises by \npj{Q3a_vs_grpo.qwen3-0.6b.mp.oracle_cov@0.2.ppabs}--\npj{Q3a_vs_grpo.phi-4-mini.mp.oracle_cov@0.2.ppabs} percentage points, and ranking gains transfer to unseen tasks. Matched controls identify proper, per-rationale scoring as the active design choices, while verification tasks show the boundary: better selective ranking can coexist with lower full-coverage accuracy. Code is available at https://github.com/ZimmyGao/openjev-rlcd
查看此版本解读 →查看此版本原文 ↗