arXiv:2604.17714 · v2
Screen Before You Interpret: A Portable Validity Screen for Benchmark-Based LLM Confidence Signals
摘要
LLM confidence signals drive abstention, routing and escalation, yet evaluations rarely check whether a signal varies with correctness before computing metrics on it. Clinical personality assessment handles the analogous problem by screening response validity before interpreting substantive scales. We specify a portable screen in that tradition. From the correctness-by-confidence table it computes the rate of high confidence on errors (L), the rate of low confidence on correct answers (Fp), and their contrast (RBS, equal to minus Youden's J). It adds no new statistics: it separates blanket confidence, blanket withdrawal and inversion, attaches interval guards, and calls a signal Valid only when it measurably tracks correctness. On 20 frontier models answering 452 two-alternative items with binary KEEP/WITHDRAW probes, one model is Invalid (DeepSeek-R1 withdraws 95% of its correct answers), four are Indeterminate and 15 are Valid. Two Indeterminate models keep confidence on 96% of their errors but make too few errors to certify it; certification needs 35 to 139 errors. Only 6 of the 20 verdicts reproduce reliably on half the items. On verbalized 0-100 confidence (33 models on 1,500 MMLU items, 20 on a second 500-item sample, and Yang et al.'s data for 11 models on 10 benchmarks), the verdict depends on the cut point: at 50 most models are Invalid because answers pile at the ceiling, while at 90-95 most pass. Battery item sensitivity does not predict MMLU discrimination (rho = -.07). Protocol amendments were settled on these data, so the verdicts are descriptive pending a held-out test. This version corrects scoring, grading, classification and reporting errors in version 1 and supersedes arXiv:2604.17707 and 2604.17716.
作者备注:v2: Corrected and consolidated; supersedes arXiv:2604.17707 and 2604.17716. Fixes errors in v1: declines were scored as confident errors, a free-text track was mis-graded, tiers did not come from the released tool, and the Yang et al. reanalysis was not computed from their data. 16 pages, 4 figures. Code and data: https://github.com/synthiumjp/validity-scaling-llm