Yiheng Li, Wenhao Wang, Xingyi Li, Zhilin Tang, Zichang Tan, Zhen Lei, Yang Yang
原文摘要
In multimodal multi-turn dialogue, dialogue history plays a dual role: reliable responses provide useful context for subsequent turns, whereas hallucinated responses can propagate errors across turns. Existing training-free methods globally adjust the effect of dialogue history on generation, making it difficult to suppress misleading information without weakening useful context. To address this trade-off, we introduce Support-Guided Triadic Decoding (SGTD), a training-free framework that selectively controls the influence of unreliable history and fuses complementary predictions during decoding. Specifically, Historical Answer Support Estimation measures the visual support of historical responses against type-matched alternatives and identifies low-support content that may mislead later generation. Support-Targeted Attention Intervention then weakens the influence of this content while preserving the surrounding dialogue, producing a support-refined prediction. Triadic Decoding combines this prediction with the full-history prediction and answer-free visual evidence, while the Support-Adaptive Plausibility Constraint filters implausible tokens using visual agreement and historical support. Experiments on MMHalSnowball and VisDiaHalBench across representative LVLMs show that SGTD consistently achieves the highest accuracy under hallucinated histories while preserving the benefits of factual context.
Yiheng Li, Wenhao Wang, Xingyi Li, Mingda Jia, Zichang Tan, Yinhang Tang, Zhen Lei, Yang Yang
原文摘要
Despite strong multimodal capabilities, large vision-language models (LVLMs) often hallucinate when linguistic priors outweigh visual evidence or when predictions are grounded in irrelevant or misleading image regions. Training-free contrastive decoding mitigates hallucinations by contrasting the original prediction with an auxiliary one that exposes linguistic priors. Yet existing methods focus on altering the auxiliary input or computation and do not ensure that either branch uses query-relevant visual evidence during decoding. We propose Visual Evidence-Guided Decoding (VEGD), a training-free framework that identifies query-relevant visual evidence and guides its use throughout contrastive decoding. VEGD locates this evidence by aggregating localization heads with strong visual responses and coherent spatial structure, then selectively weakens it in the auxiliary image. During generation, VEGD directs visual-carrier heads in both branches toward the same evidence location, strengthening access to intact evidence in the main branch while anchoring the auxiliary branch to its weakened counterpart. An adaptive distribution fusion strategy then combines or contrasts their predictions according to distribution divergence. Experiments across diverse LVLMs and benchmarks demonstrate that VEGD consistently reduces hallucinations.