arXiv:2611.01712 · v1

When Disagreement Labels Do Not Identify F1 Rankings

首次提交 2026-11-01 17:01

版本更新 2026-11-01 17:01

雷达观测 2026-11-04 23:43

摘要

Agreement labels cancel from an accuracy comparison but can change an F1 ranking. We specialize partial identification to an explicit audit after labeling disagreements: two agreement counts give exact bounds, certifying a ranking or a choice within a specified F1 tolerance under every completion. Across 3,200 frozen model pairs, shared completions tighten bounds and a 0.01 tolerance reduces labeling costs. A zero-loss (weak-winner) control attributes most savings to positive tolerance, not ties. Guarantees concern the evaluated batch, not future data.

作者备注：6 pages, 1 figure, 1 table. Submitted to ESANN 2027, submission ES2027-34. Code and reproducibility artifact: https://doi.org/10.5281/zenodo.23647323