跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2606.32034 · v2

QVal: Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

QVal:评估长时程LLM智能体的稠密监督信号

首次提交 2026-07-01 01:58
版本更新 2026-11-14 01:57
雷达观测 2026-11-16 13:44
arXiv ↗PDF ↗

摘要

LLM agents increasingly act over long horizons, where outcome-only rewards say little about which intermediate actions were good. Dense supervision methods address this by scoring intermediate steps, but they are usually evaluated through the downstream performance of a full training pipeline. This is expensive, confounds signal quality with other training choices, and means that methods are rarely compared on common ground. We introduce QVal, a training-free testbed that measures the QQ-alignment of a dense signal: whether it orders the candidate actions in each state the same way as their returns under a reference policy do. We benchmark 21 methods from seven families with six open-weight backbones in four environments. Simple prompting baselines are the most QQ-aligned on average, and performance clusters strongly by family. Code-generated signals do well only in small state and action spaces; embedding-based signals are weak; self-distillation signals are anti-aligned; and no method is QQ-aligned in open-ended environments. When five of these signals guide an actor at test time by choosing among four proposed actions, QQ-alignment strongly correlates with the gain in task success. QVal is designed to be easily extensible to new environments and methods, enabling researchers to compare new signals cheaply before committing to a training run.

作者备注:10 pages, 6 figures in main text; 67 pages, 13 figures with appendix

AI 论文解读

基于摘要 · 非全文审阅

QVal是一个无需训练的测试平台,通过衡量稠密监督信号与参考策略回报的Q对齐程度,来低成本比较不同信号的质量。

正在读取账号与研究记录…

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-14 01:57 · 最新观测版本

QVal: Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

LLM agents increasingly act over long horizons, where outcome-only rewards say little about which intermediate actions were good. Dense supervision methods address this by scoring intermediate steps, but they are usually evaluated through the downstream performance of a full training pipeline. This is expensive, confounds signal quality with other training choices, and means that methods are rarely compared on common ground. We introduce QVal, a training-free testbed that measures the QQ-alignment of a dense signal: whether it orders the candidate actions in each state the same way as their returns under a reference policy do. We benchmark 21 methods from seven families with six open-weight backbones in four environments. Simple prompting baselines are the most QQ-aligned on average, and performance clusters strongly by family. Code-generated signals do well only in small state and action spaces; embedding-based signals are weak; self-distillation signals are anti-aligned; and no method is QQ-aligned in open-ended environments. When five of these signals guide an actor at test time by choosing among four proposed actions, QQ-alignment strongly correlates with the gain in task success. QVal is designed to be easily extensible to new environments and methods, enabling researchers to compare new signals cheaply before committing to a training run.

查看此版本解读 →查看此版本原文 ↗