跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.17669 · v1

SZZBench: Controlled Git Histories with Explicit Oracles for Diagnosing SZZ Implementations

首次提交 2026-11-13 10:33
版本更新 2026-11-13 10:33
雷达观测 2026-11-17 04:57
arXiv ↗PDF ↗

摘要

The SZZ algorithm traces a bug-fixing commit back to the commits that introduced the bug. Its labels underlie defect-prediction data sets and many mining software repositories studies. SZZ implementations are mostly evaluated against oracles mined from real projects. Such oracles are realistic, but each case mixes the history factors that make an implementation succeed or fail, and an aggregate score cannot show why. We present SZZBench, a tool that shows why: it generates ordinary Git repositories from declarative scenarios with explicit oracles, scores the predictions of any SZZ implementation under declared evaluation policies, and reads each result against 26 naive baselines, such as always answering the second commit. SZZBench also checks the declarations of new scenarios, such as the tier, against the results of the naive baselines. Currently, SZZBench ships 279 scenarios in 17 categories and adapters for four SZZ tools. Running these tools in nine configurations without changing their code, we find that PyDriller passes 161 scenarios and a rule that always answers the second commit passes 160, yet only 97 scenarios are passed by both: the two succeed on different scenarios, which the number of passed scenarios hides and the per-scenario results reveal. SZZBench is available at https://github.com/higolab/SZZBench.

作者备注:5 pages, 1 figure, 2 tables

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-13 10:33 · 最新观测版本

SZZBench: Controlled Git Histories with Explicit Oracles for Diagnosing SZZ Implementations

The SZZ algorithm traces a bug-fixing commit back to the commits that introduced the bug. Its labels underlie defect-prediction data sets and many mining software repositories studies. SZZ implementations are mostly evaluated against oracles mined from real projects. Such oracles are realistic, but each case mixes the history factors that make an implementation succeed or fail, and an aggregate score cannot show why. We present SZZBench, a tool that shows why: it generates ordinary Git repositories from declarative scenarios with explicit oracles, scores the predictions of any SZZ implementation under declared evaluation policies, and reads each result against 26 naive baselines, such as always answering the second commit. SZZBench also checks the declarations of new scenarios, such as the tier, against the results of the naive baselines. Currently, SZZBench ships 279 scenarios in 17 categories and adapters for four SZZ tools. Running these tools in nine configurations without changing their code, we find that PyDriller passes 161 scenarios and a rule that always answers the second commit passes 160, yet only 97 scenarios are passed by both: the two succeed on different scenarios, which the number of passed scenarios hides and the per-scenario results reveal. SZZBench is available at https://github.com/higolab/SZZBench.

查看此版本解读 →查看此版本原文 ↗