跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.17388 · v1

Time-Consistent Evaluation of Hierarchical Code Graphs for SWE Agents

首次提交 2026-11-12 08:51
版本更新 2026-11-12 08:51
雷达观测 2026-11-17 04:58
arXiv ↗PDF ↗

摘要

Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits T0T_0 and evaluating 100 post-T0T_0 pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided. Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity (+15.6%+15.6\% to +86.2%+86.2\% relative F1F_1 lift, p<10−4p < 10^{-4}), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline F1F_1 of the next-generation model across all evaluated families.

作者备注:10 pages, 5 figures, 6 tables

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-12 08:51 · 最新观测版本

Time-Consistent Evaluation of Hierarchical Code Graphs for SWE Agents

Evaluating repository-aware software engineering (SWE) agents requires strict separation between pre-existing repository knowledge and future code evolution, yet standard benchmarks frequently suffer from temporal contamination and prompt-level solution leakage. Building on the time-consistent benchmark protocol of Sun (2026), we enforce strict zero-leakage isolation across DragonFly in C++ and React in JavaScript/TypeScript, locking repository snapshots and knowledge graphs at historical cutoff commits T0T_0 and evaluating 100 post-T0T_0 pull requests across four prompt granularities: Minimal, Concise, Contextual, and Guided. Under this zero-leakage protocol, we conduct a matched evaluation of three context-augmentation regimes: (1) an unaugmented Baseline, (2) a flat syntax-level Tree-sitter AST graph, and (3) ServiceExplorer, a deterministic two-level hierarchical dependency graph capturing Level-1 system/package containers and Level-2 architectural modules. Across 7,200 paired evaluations spanning six foundation models across the MAI-Code Flash, Claude, and GPT-Sol families, ServiceExplorer consistently outperforms both Baseline and Tree-sitter AST across every model, repository, and prompt granularity (+15.6%+15.6\% to +86.2%+86.2\% relative F1F_1 lift, p<10−4p < 10^{-4}), while tripling exact full-PR localization on DragonFly. Crucially, equipping an efficiency-tier or prior-generation model with ServiceExplorer consistently surpasses the unaugmented Baseline F1F_1 of the next-generation model across all evaluated families.

查看此版本解读 →查看此版本原文 ↗