← 研踪 PaperTrailarXiv:2605.05750 · v2
RVPO: Risk-Sensitive Alignment via Variance Regularization
首次提交 2026-05-07 14:43
版本更新 2026-10-28 03:23
雷达观测 2026-10-30 10:23
摘要
Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: strong performance in one objective (e.g., correctness) can offset critical failures in others (e.g., safety or formatting). This can mask low-performing "bottleneck" rewards that are necessary for reliable multi-objective alignment. We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes variance across rewards during advantage aggregation, and shifts the objective from "maximize sum" to "maximize consistency." Its SoftMin objective can be locally approximated by the mean reward minus an inter-objective variance penalty. We evaluate RVPO on rubric-based medical and scientific reasoning with up to 17 reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B). RVPO improves overall scores on HealthBench (0.254 vs. 0.207 for GDPO at 14B, averaged across five training seeds). RVPO also maintains competitive accuracy on GPQA-Diamond while reducing late-stage degradation observed in other multi-reward methods.
作者备注：17 pages, 5 figures
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-28 03:23 · 最新观测版本
RVPO: Risk-Sensitive Alignment via Variance Regularization
Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: strong performance in one objective (e.g., correctness) can offset critical failures in others (e.g., safety or formatting). This can mask low-performing "bottleneck" rewards that are necessary for reliable multi-objective alignment. We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes variance across rewards during advantage aggregation, and shifts the objective from "maximize sum" to "maximize consistency." Its SoftMin objective can be locally approximated by the mean reward minus an inter-objective variance penalty. We evaluate RVPO on rubric-based medical and scientific reasoning with up to 17 reward signals (Qwen2.5-3B/7B/14B) and on tool-calling with rule-based constraints (Qwen2.5-1.5B/3B). RVPO improves overall scores on HealthBench (0.254 vs. 0.207 for GDPO at 14B, averaged across five training seeds). RVPO also maintains competitive accuracy on GPQA-Diamond while reducing late-stage degradation observed in other multi-reward methods.
查看此版本解读 →查看此版本原文 ↗