跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2512.16795 · v2

From Evidence to Judgment: Trace-Augmented Reasoning under Conflicting Evidence

首次提交 2025-12-19 01:27
版本更新 2026-11-14 21:24
雷达观测 2026-11-17 12:06
arXiv ↗PDF ↗

摘要

Retrieved documents often disagree, contain outdated information, or fail to support any answer, and retrieval-augmented generation (RAG) systems have no reliable way to decide which documents to trust. We treat this as a reasoning problem and test it separately from answer generation. We release 943 training and validation examples and a disjoint 736-example benchmark, and fine-tune four language models to write a structured think trace before answering: the model judges each document, infers the conflict pattern, and writes a grounded answer or a justified refusal. Our evaluation framework, CATS (Conflict-Aware Trust Score), scores grounded refusal, behavioural adherence, factual grounding, and single-truth recall; a human study validates factual grounding and single-truth recall (kappa=0.78 and 0.79). The think trace improves the answer/refuse decision: under all three prompts, untuned models score at most 0.429 Refusal F1 (F1 with refusal as the positive class) and trace-tuned models at least 0.826. On the answerable examples a model answers, factual grounding under the 40-word prompt is at most 0.11 for untuned and 0.86-0.88 for trace-tuned models. Cutting the prompt from 1,965 words to 40 leaves trace-tuned scores within the benchmark's measurement noise (+/-0.034). Answer-only fine-tuning trails trace supervision on every metric under the minimal prompt.

作者备注:Under Review

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-14 21:24 · 最新观测版本

From Evidence to Judgment: Trace-Augmented Reasoning under Conflicting Evidence

Retrieved documents often disagree, contain outdated information, or fail to support any answer, and retrieval-augmented generation (RAG) systems have no reliable way to decide which documents to trust. We treat this as a reasoning problem and test it separately from answer generation. We release 943 training and validation examples and a disjoint 736-example benchmark, and fine-tune four language models to write a structured think trace before answering: the model judges each document, infers the conflict pattern, and writes a grounded answer or a justified refusal. Our evaluation framework, CATS (Conflict-Aware Trust Score), scores grounded refusal, behavioural adherence, factual grounding, and single-truth recall; a human study validates factual grounding and single-truth recall (kappa=0.78 and 0.79). The think trace improves the answer/refuse decision: under all three prompts, untuned models score at most 0.429 Refusal F1 (F1 with refusal as the positive class) and trace-tuned models at least 0.826. On the answerable examples a model answers, factual grounding under the 40-word prompt is at most 0.11 for untuned and 0.86-0.88 for trace-tuned models. Cutting the prompt from 1,965 words to 40 leaves trace-tuned scores within the benchmark's measurement noise (+/-0.034). Answer-only fine-tuning trails trace supervision on every metric under the minimal prompt.

查看此版本解读 →查看此版本原文 ↗