← 研踪 PaperTrailarXiv:2610.17020 · v2
CWE-bench: Refocusing Agentic Security from Exploitation to Defense
首次提交 2026-10-13 00:00
版本更新 2026-11-07 11:18
雷达观测 2026-11-10 10:11
摘要
Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An attacker needs to exploit only one vulnerability; a defender must find and fix all of them. We introduce CWE-bench, a benchmark of 250 repository audit-and-repair tasks spanning 185 CWE types and eight languages, grouped into ten weakness classes. We release 100 tasks publicly and keep 150 held out. Agents are given a repository to audit and a description of its threat model, but are not told the weakness, where it occurs, or how many vulnerable sites there are. Each patch must block the exploit while preserving original functionality, as checked by a deterministic programmatic verifier. Verifiers were curated with panels of agentic judges and reviewed by hand, so that they reject incomplete repairs without penalizing correct ones. The best agent reaches 67% pass@1 and 79% pass@4. We further conduct analyses showing that (i) agents with similar pass@1 differ widely in retry gains and cost, and (ii) no single agent is best across languages or weakness classes. As the offensive capabilities of agents improve, CWE-bench provides a broad and rigorous measure of whether their defensive capabilities keep pace.
作者备注：14 pages, 6 figures, 6 tables
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 11:18 · 最新观测版本
CWE-bench: Refocusing Agentic Security from Exploitation to Defense
Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An attacker needs to exploit only one vulnerability; a defender must find and fix all of them. We introduce CWE-bench, a benchmark of 250 repository audit-and-repair tasks spanning 185 CWE types and eight languages, grouped into ten weakness classes. We release 100 tasks publicly and keep 150 held out. Agents are given a repository to audit and a description of its threat model, but are not told the weakness, where it occurs, or how many vulnerable sites there are. Each patch must block the exploit while preserving original functionality, as checked by a deterministic programmatic verifier. Verifiers were curated with panels of agentic judges and reviewed by hand, so that they reject incomplete repairs without penalizing correct ones. The best agent reaches 67% pass@1 and 79% pass@4. We further conduct analyses showing that (i) agents with similar pass@1 differ widely in retry gains and cost, and (ii) no single agent is best across languages or weakness classes. As the offensive capabilities of agents improve, CWE-bench provides a broad and rigorous measure of whether their defensive capabilities keep pace.
查看此版本解读 →查看此版本原文 ↗