arXiv:2610.31090 · v2
Poolwise: Evaluating Reward Models for Selection across Prompts
摘要
Reward models are used to select training data, but common benchmarks compare responses only within the same prompt. Selecting data across prompts also depends on how scores compare between prompts. We introduce Poolwise, a protocol that selects a fixed fraction of a response pool by score, measures the selected set's quality using external labels, and reports within-prompt ranking separately. In matched experiments on two rating sets and two model sizes, training each prompt's mean score to match its average response rating improves selection over permuted or omitted mean targets in every seed at 30 percent retention. In grouped HelpSteer experiments, the model with the highest within-prompt accuracy selects a lower-rated set than the correct-mean model. Subtracting prompt means estimated from separate reference responses preserves local rankings but lowers selected helpfulness for six open reward models and two closed judges on HelpSteer. Poolwise provides a direct way to compare reward models and score transformations for data selection.
作者备注：38 pages, 9 figures, 47 tables. v2: reorganized around Poolwise, a protocol for evaluating reward models on pooled selection across prompts; adds matched training on HelpSteer's training split at two model sizes, two closed judges, a third reference generator and external panels (RoSE, IFEval); revised abstract, introduction and conclusion