跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.11673 · v1

Known Good, Known Bad: A Ground-Truth Calibration Protocol for AI Judgment in Science

首次提交 2026-11-09 06:59
版本更新 2026-11-09 06:59
雷达观测 2026-11-10 13:19
arXiv ↗PDF ↗

摘要

Scientific use of generative AI requires evaluation against cases with independently checkable answers. We present a ground truth calibration protocol that compares a verified reference case with reference deviant variants through neutral evaluation, an authority framed recheck, and a line by line audit. Human adjudication distinguishes deviation detection, premise accommodation, unsupported doubt, and correctness of the resulting calculation. An instructor generated organic chemistry demonstration comprises six conversations and 18 responses from an unidentified backend served through the free logged out ChatGPT interface in July 2026. Neutral evaluation detected three of five intended deviations. Both initial misses, involving oxidant stoichiometry and experimental scale, were detected at the authority framed second turn; all five detections persisted at the third turn audit. The fixed sequence supports a descriptive account of recovery across turns. In the scale case, the model initially produced a locally coherent peroxide limited yield. After recognizing the unit problem, it incorrectly retained that limiting reagent conclusion for the corrected scale. Detection and numerical repair therefore require separate assessment. The verified reference attracted speculative concerns while retaining its correct yield calculation. The protocol connects documentary verification of scientific claims and citations with empirical verification of laboratory interpretations. It supplies inspectable transcripts, reference calculations, behavior codes, and a reusable calibration architecture for AI literacy instruction. The pilot demonstrates the protocol's measures in one case family; replicated model evaluation and learner assessment are the next validation steps.

作者备注:18 pages, 3 figures, 6 tables

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-09 06:59 · 最新观测版本

Known Good, Known Bad: A Ground-Truth Calibration Protocol for AI Judgment in Science

Scientific use of generative AI requires evaluation against cases with independently checkable answers. We present a ground truth calibration protocol that compares a verified reference case with reference deviant variants through neutral evaluation, an authority framed recheck, and a line by line audit. Human adjudication distinguishes deviation detection, premise accommodation, unsupported doubt, and correctness of the resulting calculation. An instructor generated organic chemistry demonstration comprises six conversations and 18 responses from an unidentified backend served through the free logged out ChatGPT interface in July 2026. Neutral evaluation detected three of five intended deviations. Both initial misses, involving oxidant stoichiometry and experimental scale, were detected at the authority framed second turn; all five detections persisted at the third turn audit. The fixed sequence supports a descriptive account of recovery across turns. In the scale case, the model initially produced a locally coherent peroxide limited yield. After recognizing the unit problem, it incorrectly retained that limiting reagent conclusion for the corrected scale. Detection and numerical repair therefore require separate assessment. The verified reference attracted speculative concerns while retaining its correct yield calculation. The protocol connects documentary verification of scientific claims and citations with empirical verification of laboratory interpretations. It supplies inspectable transcripts, reference calculations, behavior codes, and a reusable calibration architecture for AI literacy instruction. The pilot demonstrates the protocol's measures in one case family; replicated model evaluation and learner assessment are the next validation steps.

查看此版本解读 →查看此版本原文 ↗