← 研踪 PaperTrailarXiv:2606.32034 · v2
QVal: Evaluating Dense Supervision Signals for Long-Horizon LLM Agents
QVal:评估长时程LLM智能体的稠密监督信号
首次提交 2026-07-01 01:58
版本更新 2026-11-14 01:57
雷达观测 2026-11-16 13:44
摘要
LLM agents increasingly act over long horizons, where outcome-only rewards say little about which intermediate actions were good. Dense supervision methods address this by scoring intermediate steps, but they are usually evaluated through the downstream performance of a full training pipeline. This is expensive, confounds signal quality with other training choices, and means that methods are rarely compared on common ground. We introduce QVal, a training-free testbed that measures the -alignment of a dense signal: whether it orders the candidate actions in each state the same way as their returns under a reference policy do. We benchmark 21 methods from seven families with six open-weight backbones in four environments. Simple prompting baselines are the most -aligned on average, and performance clusters strongly by family. Code-generated signals do well only in small state and action spaces; embedding-based signals are weak; self-distillation signals are anti-aligned; and no method is -aligned in open-ended environments. When five of these signals guide an actor at test time by choosing among four proposed actions, -alignment strongly correlates with the gain in task success. QVal is designed to be easily extensible to new environments and methods, enabling researchers to compare new signals cheaply before committing to a training run.
作者备注:10 pages, 6 figures in main text; 67 pages, 13 figures with appendix
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-14 01:57 · 最新观测版本
QVal: Evaluating Dense Supervision Signals for Long-Horizon LLM Agents
LLM agents increasingly act over long horizons, where outcome-only rewards say little about which intermediate actions were good. Dense supervision methods address this by scoring intermediate steps, but they are usually evaluated through the downstream performance of a full training pipeline. This is expensive, confounds signal quality with other training choices, and means that methods are rarely compared on common ground. We introduce QVal, a training-free testbed that measures the -alignment of a dense signal: whether it orders the candidate actions in each state the same way as their returns under a reference policy do. We benchmark 21 methods from seven families with six open-weight backbones in four environments. Simple prompting baselines are the most -aligned on average, and performance clusters strongly by family. Code-generated signals do well only in small state and action spaces; embedding-based signals are weak; self-distillation signals are anti-aligned; and no method is -aligned in open-ended environments. When five of these signals guide an actor at test time by choosing among four proposed actions, -alignment strongly correlates with the gain in task success. QVal is designed to be easily extensible to new environments and methods, enabling researchers to compare new signals cheaply before committing to a training run.
查看此版本解读 →查看此版本原文 ↗