← 研踪 PaperTrailarXiv:2605.06919 · v2
Can LLMs Take Retrieved Information with a Grain of Salt?
首次提交 2026-05-08 04:29
版本更新 2026-11-07 01:37
雷达观测 2026-11-09 13:22
摘要
Large language models have demonstrated impressive retrieval-augmented capabilities. However, a crucial area remains underexplored: their ability to appropriately adapt responses to the certainty of the retrieved information. It is a limitation with real consequences in high-stakes domains like medicine and finance. We evaluate eight LLMs on their context-certainty obedience, measuring how well they adjust responses to match expressed context certainty. Our analysis reveals systematic limitations: LLMs fail to reproduce their prior outputs after observing an uncertain context, respond to expressed certainties in a distorted way, and overrely on complex contexts. To address these, we propose an interaction strategy combining prior reminders (feeding the model's own context-free answer back into the prompt), context-certainty recalibration, and context simplification. This approach reduces obedience errors by 25% on average, without modifying model weights, demonstrating the efficacy of interaction design in enhancing LLM reliability. Our contributions include a principled evaluation metric, empirical insights into LLMs' uncertainty handling, and a portable strategy to improve context-certainty obedience across diverse LLMs.
作者备注：Accepted by The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS 2026)
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 01:37 · 最新观测版本
Can LLMs Take Retrieved Information with a Grain of Salt?
Large language models have demonstrated impressive retrieval-augmented capabilities. However, a crucial area remains underexplored: their ability to appropriately adapt responses to the certainty of the retrieved information. It is a limitation with real consequences in high-stakes domains like medicine and finance. We evaluate eight LLMs on their context-certainty obedience, measuring how well they adjust responses to match expressed context certainty. Our analysis reveals systematic limitations: LLMs fail to reproduce their prior outputs after observing an uncertain context, respond to expressed certainties in a distorted way, and overrely on complex contexts. To address these, we propose an interaction strategy combining prior reminders (feeding the model's own context-free answer back into the prompt), context-certainty recalibration, and context simplification. This approach reduces obedience errors by 25% on average, without modifying model weights, demonstrating the efficacy of interaction design in enhancing LLM reliability. Our contributions include a principled evaluation metric, empirical insights into LLMs' uncertainty handling, and a portable strategy to improve context-certainty obedience across diverse LLMs.
查看此版本解读 →查看此版本原文 ↗