← 研踪 PaperTrailarXiv:2604.07789 · v4
ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents
首次提交 2026-04-09 12:37
版本更新 2026-11-14 20:13
雷达观测 2026-11-17 12:06
摘要
Recent advances in Language Model (LM) agents have increasingly focused on repository-level software engineering (SWE) tasks, leading to various agentic workflows, training methods and failure analysis. Interestingly, rather than directly generating or evaluating edit patches, these approaches first examine and improve intermediate information signals collected during agent exploration, including Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage. Although different studies examine different subsets of these signals, no work has systematically compared their relative effectiveness in improving final task success or measured the remaining space for improvement in each signal. To address this gap, this work quantifies the contributions of both oracle and agent-extracted signals to final task success and measures the gap between current agent-found signals and their oracle counterparts. We hope our findings could suggest future research emphasis in SWE agents. We release our source code and oracle-signal dataset as a reusable methodology to benefit future signal analysis of SWE agents.
作者备注:EACL 2027 Findings
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v4 · 2026-11-14 20:13 · 最新观测版本
ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents
Recent advances in Language Model (LM) agents have increasingly focused on repository-level software engineering (SWE) tasks, leading to various agentic workflows, training methods and failure analysis. Interestingly, rather than directly generating or evaluating edit patches, these approaches first examine and improve intermediate information signals collected during agent exploration, including Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage. Although different studies examine different subsets of these signals, no work has systematically compared their relative effectiveness in improving final task success or measured the remaining space for improvement in each signal. To address this gap, this work quantifies the contributions of both oracle and agent-extracted signals to final task success and measures the gap between current agent-found signals and their oracle counterparts. We hope our findings could suggest future research emphasis in SWE agents. We release our source code and oracle-signal dataset as a reusable methodology to benefit future signal analysis of SWE agents.
查看此版本解读 →查看此版本原文 ↗