Constructing a Japanese Verdict Prediction Dataset for Fact-Checking of LLM-Generated Texts
Miwa Masano, Hirokazu Kiyomaru National Institute of Informatics, Atsushi Keyaki Hitotsubashi University, Kaito Horio Waseda University, Rei Minamoto, Ribeka Keyaki Tokyo University of Technology, Kouta Nakayama NII, Tokyo Institute of Technology, Hideyuki Tachibana National Institute of Informatics and Asia University 等 9 位作者
原文摘要
The development of fact-checking systems for verifying the factuality of text generated by large language models (LLMs) has been advancing.In the verdict prediction step of such systems, the system determines whether claims in the generated text are supported by retrieved evidence, formulated as a natural language inference (NLI) task.This study extends the label set for verdict prediction to capture claim-evidence relationships that humans would commonly interpret as supported or refuted, even in the absence of strict logical entailment or contradiction.It also constructs a Japanese dataset comprising 28,147 instances from two sources based on this extended label set.We analyze the causes of annotation disagreement and find that ambiguity in the boundary of acceptable inference, interpretive characteristics of negative cases, and incomplete information in the evidence affect annotation variability.Using this dataset, we evaluate the performance of prompt-based verdict prediction methods and show that prompts that explicitly elicit chain-of-thought reasoning improve F1 by 4 percentage points compared to baseline.
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) · 2026 · 正式发表