跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.25924 · v1

Ground Truth Bias in AI-Assisted Medical Annotation: A Crossover Study Using an Independent Radiologist Reference Standard

首次提交 2026-11-19 19:08
版本更新 2026-11-19 19:08
雷达观测 2026-11-20 19:57
arXiv ↗PDF ↗

摘要

Behavioural evaluations of artificial-intelligence-assisted annotation score every human decision against a reference standard, so their conclusions inherit that standard's validity. We tested how far conclusions about assistance, reliance and individual differences depend on it. Fifty-one non-expert high-school students annotated 50 knee radiographs for osteoarthritis, with and without a deliberately imperfect classifier, in a crossover design; personality and non-verbal reasoning were also measured. Decisions were scored against the public repository labels and against an independent reference standard from a radiologist panel, which graded 23 of the 50 images as indeterminate. Against the repository labels, assistance raised accuracy from 60.4% to 65.3% (odds ratio 1.23, p=0.002). Against the independent standard, the gain on the 27 determinate images was smaller and not significant in the primary model (78.4% to 80.8%; odds ratio 1.16, p=0.178). The apparent benefit had been concentrated on the indeterminate images and was not significant when those images were assigned to either class. The direction of residual over-reliance depended on how indeterminate cases were handled, and the personality trait most strongly associated with accuracy changed identity and sign between standards. Reliance measures are only as valid as the reference standard that defines when the assistant is right.

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-19 19:08 · 最新观测版本

Ground Truth Bias in AI-Assisted Medical Annotation: A Crossover Study Using an Independent Radiologist Reference Standard

Behavioural evaluations of artificial-intelligence-assisted annotation score every human decision against a reference standard, so their conclusions inherit that standard's validity. We tested how far conclusions about assistance, reliance and individual differences depend on it. Fifty-one non-expert high-school students annotated 50 knee radiographs for osteoarthritis, with and without a deliberately imperfect classifier, in a crossover design; personality and non-verbal reasoning were also measured. Decisions were scored against the public repository labels and against an independent reference standard from a radiologist panel, which graded 23 of the 50 images as indeterminate. Against the repository labels, assistance raised accuracy from 60.4% to 65.3% (odds ratio 1.23, p=0.002). Against the independent standard, the gain on the 27 determinate images was smaller and not significant in the primary model (78.4% to 80.8%; odds ratio 1.16, p=0.178). The apparent benefit had been concentrated on the indeterminate images and was not significant when those images were assigned to either class. The direction of residual over-reliance depended on how indeterminate cases were handled, and the personality trait most strongly associated with accuracy changed identity and sign between standards. Reliance measures are only as valid as the reference standard that defines when the assistant is right.

查看此版本解读 →查看此版本原文 ↗