跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.12820 · v1

How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors

AI 同行评审效果如何?一个基于100个植入错误的基准

首次提交 2026-11-10 02:22
版本更新 2026-11-10 02:22
雷达观测 2026-11-10 15:22
arXiv ↗PDF ↗

摘要

One hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools. The best single system caught 71 of the 100 errors, while the worst caught 30. Pooling every system's output caught 93. Models are only partly correlated in the errors they find, which makes ensembling a big lever for finding issues in papers. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. Refine.ink contributes more unique catches than any other single system, though it is expensive on a per-error basis. False positives were not measured, and how this error distribution compares to the distribution of errors in real papers remains unknown. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.

作者备注:5 pages; accepted at Cognitive Security for Human-AI Systems (CogSec 2026), a non-archival workshop at ACM HCOMP / CI 2026

AI 论文解读

基于摘要 · 非全文审阅

作者将100个已知错误植入10篇开放获取心理学论文,用前沿模型和两款商业AI审稿工具检测,并比较单系统与集成系统的查错表现。

正在读取账号与研究记录…

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-10 02:22 · 最新观测版本

How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors

One hundred known errors were planted into 10 open-access psychology papers, and the modified papers were then run through frontier models and two commercial AI review tools. The best single system caught 71 of the 100 errors, while the worst caught 30. Pooling every system's output caught 93. Models are only partly correlated in the errors they find, which makes ensembling a big lever for finding issues in papers. Seven errors could not be caught by any system, and all seven were omissions: information deleted from a paper rather than mistakes inserted into it. Refine.ink contributes more unique catches than any other single system, though it is expensive on a per-error basis. False positives were not measured, and how this error distribution compares to the distribution of errors in real papers remains unknown. The papers, errors, model outputs, and full experiment log are public, so that others can build on this toward a comprehensive evaluation benchmark across disciplines.

查看此版本解读 →查看此版本原文 ↗