Build-and-Find:一种评估智能体管理代码库的努力感知协议
Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
Jhen-Ke Lin
AI 速读 基于摘要
提出 Build-and-Find 协议,将代码库视为面向后续智能体的沟通产物,用恢复准确率、稳定性与检查努力等指标评估生成仓库的可读性与意图可恢复性。
cs.SE · cs.AI
查看原文摘要
Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce Build-and-Find to evaluate whether downstream agents can efficiently recover intended choices from generated repositories. For each task, a builder creates a codebase from a hidden specification; a finder receives only the codebase and a specification-traced question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Evaluation proceeds in two stages: the first uses recovery accuracy and stability as gates to establish that finders can reliably recover the intended choices; the second compares inspection effort by the same finder across artifacts that pass these gates and allow the same intent to be recovered. Lower effort suggests that the artifact makes that intent easier to locate. We find that several frontier models approach saturation in intent recovery on our task pack, yet second-stage inspection effort reveals differences among their generated artifacts, providing a dimension beyond accuracy for comparing artifact legibility. We also audit artifacts to distinguish unimplemented specification claims from finder failures and check whether correct answers are supported by artifact evidence. We release the harness and benchmark data for reproducibility and extension.