← 研踪 PaperTrailarXiv:2607.00516 · v3
When A Beats B Needs Evidence: CLAIMSTAB-QC for Auditing Comparative Claims in Quantum Software
首次提交 2026-07-01 15:02
版本更新 2026-11-05 18:52
雷达观测 2026-11-06 12:09
摘要
Empirical quantum-software papers often claim that one compiler produces smaller circuits, one optimizer is more reliable, or one ansatz achieves better solution quality. Such comparisons depend on benchmark scope, circuit construction, compilation, sampling, backends, noise models, optimizer choices, and resource budgets. Existing testing, benchmarking, and reproducibility methods help assess programs and experiments but do not directly audit whether the reported comparison is supported by the evidence available in the paper and artifact. We present CLAIMSTAB-QC, a source-bounded framework that records baselines, metric, relation, and admissible evidence; locks the audit design before outcomes are computed; and reports a scoped relation outcome or an explicit evidence boundary. Strict scalar-directional comparisons are classified as Sustained, Unresolved, or Not preserved within the locked audit scope. We examine 455 comparative claims from 119 quantum-software papers. A materialization gap emerges: 175 claims can be represented for audit planning, 79 become scalar-directional planning records, and 53 yield lockable audit or diagnostic designs, but only 8 expose enough matched evidence to audit the original comparison without proxy reconstruction. Among these 8, 2 are Sustained, 4 Unresolved, and 2 Not preserved. Controlled diagnostics over 24 benchmark-relevant quantum-workload comparisons further show that simpler checks can preserve apparent directions despite weaker support under locked designs. These results indicate that performance orderings should be reported with matched evidence sufficient to audit their stated scope.
作者备注：12 pages, 4 figures, 5 tables
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-05 18:52 · 最新观测版本
When A Beats B Needs Evidence: CLAIMSTAB-QC for Auditing Comparative Claims in Quantum Software
Empirical quantum-software papers often claim that one compiler produces smaller circuits, one optimizer is more reliable, or one ansatz achieves better solution quality. Such comparisons depend on benchmark scope, circuit construction, compilation, sampling, backends, noise models, optimizer choices, and resource budgets. Existing testing, benchmarking, and reproducibility methods help assess programs and experiments but do not directly audit whether the reported comparison is supported by the evidence available in the paper and artifact. We present CLAIMSTAB-QC, a source-bounded framework that records baselines, metric, relation, and admissible evidence; locks the audit design before outcomes are computed; and reports a scoped relation outcome or an explicit evidence boundary. Strict scalar-directional comparisons are classified as Sustained, Unresolved, or Not preserved within the locked audit scope. We examine 455 comparative claims from 119 quantum-software papers. A materialization gap emerges: 175 claims can be represented for audit planning, 79 become scalar-directional planning records, and 53 yield lockable audit or diagnostic designs, but only 8 expose enough matched evidence to audit the original comparison without proxy reconstruction. Among these 8, 2 are Sustained, 4 Unresolved, and 2 Not preserved. Controlled diagnostics over 24 benchmark-relevant quantum-workload comparisons further show that simpler checks can preserve apparent directions despite weaker support under locked designs. These results indicate that performance orderings should be reported with matched evidence sufficient to audit their stated scope.
查看此版本解读 →查看此版本原文 ↗v2 · 2026-11-04 17:02
Auditing Empirical Comparisons in Quantum Software
Empirical quantum-software papers often claim that one compiler produces smaller circuits, one optimizer is more reliable, or one ansatz achieves better solution quality. Such comparisons depend on benchmark scope, circuit construction, compilation, sampling, backends, noise models, optimizer choices, and resource budgets. Existing testing, benchmarking, and reproducibility methods help assess programs and experiments but do not directly audit whether the reported comparison is supported by the evidence available in the paper and artifact. We present CLAIMSTAB-QC, a source-bounded framework that records baselines, metric, relation, and admissible evidence; locks the audit design before outcomes are computed; and reports a scoped relation outcome or an explicit evidence boundary. Strict scalar-directional comparisons are classified as Sustained, Unresolved, or Not preserved within the locked audit scope. We examine 455 comparative claims from 119 quantum-software papers. A materialization gap emerges: 175 claims can be represented for audit planning, 79 become scalar-directional planning records, and 53 yield lockable audit or diagnostic designs, but only 8 expose enough matched evidence to audit the original comparison without proxy reconstruction. Among these 8, 2 are Sustained, 4 Unresolved, and 2 Not preserved. Controlled diagnostics over 24 benchmark-relevant quantum-workload comparisons further show that simpler checks can preserve apparent directions despite weaker support under locked designs. These results indicate that performance orderings should be reported with matched evidence sufficient to audit their stated scope.
查看此版本解读 →查看此版本原文 ↗