← 研踪 PaperTrailarXiv:2605.26321 · v3
Anchor: Preventing Artifact Drift for Reliable Agent Verification
首次提交 2026-05-26 04:44
版本更新 2026-11-16 02:27
雷达观测 2026-11-17 17:12
摘要
Reliable evaluation of long-horizon agents requires verifiers that match what the task actually asks. Verifiable rewards have driven progress in domains like math and coding, yet building similarly reliable environments for open-ended enterprise workflows repeatedly struggles to balance realism, verifiability, and scale. This creates a validity problem we call artifact drift: when instructions, environments, oracles, and verifiers are created separately, they frequently fail to agree on what a task requires, resulting in environments that are unsolvable, reward-hackable, or inconsistent. We introduce Anchor, a benchmark-construction pipeline that formalizes domain expert specifications of business workflows into constrained optimization programs. From a single parametric specification, the pipeline jointly produces a natural-language instruction, environment configuration, solver-certified ground-truth solution, and state-based verifier. With this pipeline, altering parameters yields new tasks with controlled difficulty and known optimal solutions, producing harness-agnostic environments whose rewards depend solely on end-state business correctness. We apply Anchor to produce ERP-Bench: a benchmark of 300 long-horizon tasks spanning procurement and manufacturing workflows in a production-grade ERP system. Our pipeline organizes tasks into diverse families with calibrated difficulty profiles, and allows generated environments to evaluate coding, browser, and computer-use agents using a shared verifier. Across these harnesses, we find that frontier models follow all rules and instructions in 25.9% of trials and reach fully optimal decisions in only 17.4% of trials. Overall, Anchor and ERP-Bench offer a concrete recipe for building auditable evaluation environments whose verifiers remain aligned with the economically valuable tasks they are intended to score.
作者备注:Accepted to the NeurIPS 2026 Workshop on Who Verifies the Agents?
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-16 02:27 · 最新观测版本
Anchor: Preventing Artifact Drift for Reliable Agent Verification
Reliable evaluation of long-horizon agents requires verifiers that match what the task actually asks. Verifiable rewards have driven progress in domains like math and coding, yet building similarly reliable environments for open-ended enterprise workflows repeatedly struggles to balance realism, verifiability, and scale. This creates a validity problem we call artifact drift: when instructions, environments, oracles, and verifiers are created separately, they frequently fail to agree on what a task requires, resulting in environments that are unsolvable, reward-hackable, or inconsistent. We introduce Anchor, a benchmark-construction pipeline that formalizes domain expert specifications of business workflows into constrained optimization programs. From a single parametric specification, the pipeline jointly produces a natural-language instruction, environment configuration, solver-certified ground-truth solution, and state-based verifier. With this pipeline, altering parameters yields new tasks with controlled difficulty and known optimal solutions, producing harness-agnostic environments whose rewards depend solely on end-state business correctness. We apply Anchor to produce ERP-Bench: a benchmark of 300 long-horizon tasks spanning procurement and manufacturing workflows in a production-grade ERP system. Our pipeline organizes tasks into diverse families with calibrated difficulty profiles, and allows generated environments to evaluate coding, browser, and computer-use agents using a shared verifier. Across these harnesses, we find that frontier models follow all rules and instructions in 25.9% of trials and reach fully optimal decisions in only 17.4% of trials. Overall, Anchor and ERP-Bench offer a concrete recipe for building auditable evaluation environments whose verifiers remain aligned with the economically valuable tasks they are intended to score.
查看此版本解读 →查看此版本原文 ↗