← 研踪 PaperTrailarXiv:2605.06136 · v2
Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
首次提交 2026-05-07 20:35
版本更新 2026-11-07 02:16
雷达观测 2026-11-09 13:22
摘要
Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce Build-and-Find to evaluate whether downstream agents can efficiently recover intended choices from generated repositories. For each task, a builder creates a codebase from a hidden specification; a finder receives only the codebase and a specification-traced question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Evaluation proceeds in two stages: the first uses recovery accuracy and stability as gates to establish that finders can reliably recover the intended choices; the second compares inspection effort by the same finder across artifacts that pass these gates and allow the same intent to be recovered. Lower effort suggests that the artifact makes that intent easier to locate. We find that several frontier models approach saturation in intent recovery on our task pack, yet second-stage inspection effort reveals differences among their generated artifacts, providing a dimension beyond accuracy for comparing artifact legibility. We also audit artifacts to distinguish unimplemented specification claims from finder failures and check whether correct answers are supported by artifact evidence. We release the harness and benchmark data for reproducibility and extension.
作者备注：36 pages, 8 figures. Accepted at NeurIPS 2026. Code: https://github.com/JacobLinCool/Build-and-Find
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 02:16 · 最新观测版本
Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce Build-and-Find to evaluate whether downstream agents can efficiently recover intended choices from generated repositories. For each task, a builder creates a codebase from a hidden specification; a finder receives only the codebase and a specification-traced question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Evaluation proceeds in two stages: the first uses recovery accuracy and stability as gates to establish that finders can reliably recover the intended choices; the second compares inspection effort by the same finder across artifacts that pass these gates and allow the same intent to be recovered. Lower effort suggests that the artifact makes that intent easier to locate. We find that several frontier models approach saturation in intent recovery on our task pack, yet second-stage inspection effort reveals differences among their generated artifacts, providing a dimension beyond accuracy for comparing artifact legibility. We also audit artifacts to distinguish unimplemented specification claims from finder failures and check whether correct answers are supported by artifact evidence. We release the harness and benchmark data for reproducibility and extension.
查看此版本解读 →查看此版本原文 ↗