The Imperfective Paradox Is Not Necessarily in Large Language Models: A Benchmark Failure Before a Model Failure
Kaiqiao Han, Katarzyna Pruś, Mark Steedman, Yizhou Sun
原文摘要
Recent work uses an NLI benchmark of the imperfective paradox to argue that large language models exhibit a Teleological Bias and Calibration Crisis. We identify three validity concerns in benchmark construction, including aspectual reduction, semantic bias, and relation bias. Under a strict NLI standard, 76% of the Group A instances do not explicitly rule out culmination; in native-speaker annotation, 38% of Group A and 29% of Group C examples permit an alternative interpretation. To better isolate the semantic contrast, we construct Lexically Matched Minimal Pairs. Moreover, by formulating event-semantic NLI as a multi-step reasoning problem, we evaluate intermediate semantic judgments and use oracle-guided settings. We find that incorrect NLI entailment does not fully support teleological bias. Prompting often changes final NLI decisions without reliably improving the underlying semantic judgments, offering a possible explanation for the Calibration Crisis.
cs.CL · cs.LG