arXiv:2608.00794 · v5
Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
摘要
Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims. No formal framework has characterized how validity degrades across these stages. We present a knowledge-based three-layer compounding validity model, $V_{\rm total} \leq V_1 \times V_2 \times V_3$, that captures multiplicative degradation across task generation ($V_1$), human-simulator calibration ($V_2$), and automated judgment ($V_3$). Applying it to five widely-used agentic benchmark configurations, we find $V_1$ and $V_2$ independently measurable and both degraded, but $V_3$ not currently quantifiable for any: three of the four benchmarks score with a single deterministic rule, so no second rater exists and reliability is undefined, and the fourth's partial LLM grader has no published grader-vs-human measurement. A structured survey of 55 published agentic evaluation papers finds that 26 of them report no chance-corrected inter-rater statistic of any kind, a determination four independent raters recover at Krippendorff's $α= 0.834$. On the broader judgment of whether a reported metric fits its rating design, approximately 80% of this purposive sample use structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics (four-rater range 60.7%-84.4%). Both are consistent with systematic $V_3$ collapse. We further document $V_1$ failures (task validity flaws in 7 of 10 benchmarks) and $V_2$ miscalibration (up to 9 percentage points inter-simulator variance, with systematic demographic disparities for non-Standard American English speakers). We derive eight prescriptions grounded in psychometric science and domain-stratified reliability thresholds (intraclass correlation $\geq 0.70$; $α\geq 0.67/0.70/0.80$ by consequence level) that practitioners can apply immediately.
作者备注：50 pages, 2 figures, 7 tables. Second revision (R2) under review at Knowledge-Based Systems (Elsevier). Code and data: https://github.com/williamcaban/experiment-measurement-without-validity