Observational Equivalence of LLM and Human Annotation
Kentaro Nakamura, Jing Ling Tan, George Yean
In this paper, we show that humans and large language models (LLMs) are observationally equivalent in coding texts. Recent LLMs agree with attentive humans at rates comparable to those observed among humans, and cross-LLM disagreement concentrates on texts that also divide humans. We demonstrate these patterns through replications of text-classification tasks from 14 peer-reviewed political science studies, with 13 LLMs, three human coders (Ph.D. students), and 165 crowd coders, independently applying identical codebooks. We further find that clarifying an underspecified codebook increases agreement among humans and between humans and LLMs. Together, these findings point to ambiguity in texts and coding rules as a shared source of disagreement and challenge the presumption that human labels should take precedence whenever humans and LLMs differ. We therefore argue that the central challenge of text annotation is not simply choosing between humans and LLMs, but developing coding rules that minimize ambiguity and accounting for the ambiguity that remains. To this end, we propose using cross-LLM disagreement to identify difficult cases, guide human review, and refine codebooks, and we develop ambiguity-aware bounds for downstream inference without requiring ground truth for every text.
尚无 AI 解读，展示原文摘要