← 研踪 PaperTrailarXiv:2505.04311 · v2
How the Misuse of a Dataset Harmed Semantic Clone Detection
首次提交 2025-05-07 18:52
版本更新 2026-11-04 18:15
雷达观测 2026-11-05 12:13
摘要
BigCloneBench is widely used to evaluate traditional clone detection and, more recently, machine learning approaches to semantic clone detection and functional similarity. This paper demonstrates that BigCloneBench's Weakly Type-3/Type-4 (WT3/T4) subset is problematic as ground truth for learning or evaluating functional similarity. Its tagged-tagged WT3/T4 clone pairs combine two methods labelled as true positives for the same functionality and do not contain exemplar functions. A population-weighted analysis of a stratified probability sample of 406 such pairs estimated that 93.9% of pairs in the eligible tagged-tagged WT3/T4 population are not functionally similar under our operational definition and are therefore mislabelled (conservative 95% confidence interval: 91.5%-96.2%). From a literature review of 179 papers, we estimate that 128 used BigCloneBench to evaluate semantic clone detection in a way that their results could be affected by the mislabelling. High F1 scores on BigCloneBench do not establish whether an approach has learned functional similarity or relied on dataset-specific shortcuts. Our study does not assess BigCloneBench's validity for evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones. We acknowledge its important contributions to traditional clone detection research. However, using its WT3/T4 subset as ground truth for same-main-function similarity without validating its labels for that task constitutes misuse. By threatening the validity of a substantial body of reported results, this practice has harmed the evidence base of semantic clone detection. We encourage more critical and responsible use of benchmarks and datasets and more rigorous validation during dataset creation and use.
作者备注：60 pages. Accepted for publication in Empirical Software Engineering
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-04 18:15 · 最新观测版本
How the Misuse of a Dataset Harmed Semantic Clone Detection
BigCloneBench is widely used to evaluate traditional clone detection and, more recently, machine learning approaches to semantic clone detection and functional similarity. This paper demonstrates that BigCloneBench's Weakly Type-3/Type-4 (WT3/T4) subset is problematic as ground truth for learning or evaluating functional similarity. Its tagged-tagged WT3/T4 clone pairs combine two methods labelled as true positives for the same functionality and do not contain exemplar functions. A population-weighted analysis of a stratified probability sample of 406 such pairs estimated that 93.9% of pairs in the eligible tagged-tagged WT3/T4 population are not functionally similar under our operational definition and are therefore mislabelled (conservative 95% confidence interval: 91.5%-96.2%). From a literature review of 179 papers, we estimate that 128 used BigCloneBench to evaluate semantic clone detection in a way that their results could be affected by the mislabelling. High F1 scores on BigCloneBench do not establish whether an approach has learned functional similarity or relied on dataset-specific shortcuts. Our study does not assess BigCloneBench's validity for evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones. We acknowledge its important contributions to traditional clone detection research. However, using its WT3/T4 subset as ground truth for same-main-function similarity without validating its labels for that task constitutes misuse. By threatening the validity of a substantial body of reported results, this practice has harmed the evidence base of semantic clone detection. We encourage more critical and responsible use of benchmarks and datasets and more rigorous validation during dataset creation and use.
查看此版本解读 →查看此版本原文 ↗