← 研踪 PaperTrailarXiv:2608.12571 · v2
Is this Citation on Point?
首次提交 2026-08-13 04:28
版本更新 2026-10-26 06:54
雷达观测 2026-10-29 00:36
摘要
In 2023, a New York judge sanctioned two attorneys in \emph{Mata v. Avianca} for filing a brief containing hallucinated citations generated by ChatGPT. These failures can be largely caught by database lookups. The harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered, a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora. We either replace the cited case or change only the pinpoint page within the same case. Our evaluation of fourteen model configurations on the resulting examples shows that models catch $93$--$100\%$ of wrong-case corruptions, but they only catch $37$--$61\%$ of wrong-pinpoint corruptions on court opinions and $52$--$83\%$ on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Larger-scale models and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses $40\%$ of pinpoint mismatches on court opinions and $18\%$ on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. These results show that citation misrepresentations remain difficult to detect automatically. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.
作者备注：Accepted to the 1st Workshop on AI for Law at ICML 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-26 06:54 · 最新观测版本
Is this Citation on Point?
In 2023, a New York judge sanctioned two attorneys in \emph{Mata v. Avianca} for filing a brief containing hallucinated citations generated by ChatGPT. These failures can be largely caught by database lookups. The harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered, a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora. We either replace the cited case or change only the pinpoint page within the same case. Our evaluation of fourteen model configurations on the resulting examples shows that models catch $93$--$100\%$ of wrong-case corruptions, but they only catch $37$--$61\%$ of wrong-pinpoint corruptions on court opinions and $52$--$83\%$ on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Larger-scale models and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses $40\%$ of pinpoint mismatches on court opinions and $18\%$ on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. These results show that citation misrepresentations remain difficult to detect automatically. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.
查看此版本解读 →查看此版本原文 ↗