HalluProbe: Exploiting VLM Hallucinations for Backdoor Detection
Ji Guo, Xiaolong Qin, Jin Jiang, Yijing Liu, Pei Ke, Aiguo Chen, Wenbo Jiang
原文摘要
Vision-Language Models (VLMs) are vulnerable to backdoor attacks introduced through data poisoning, making effective backdoor detection a critical security challenge. In this paper, we uncover a novel connection between VLM hallucinations and backdoor behaviors and exploit this connection for backdoor detection. Our key insight is that poisoned training data induces strong language priors toward the backdoor target token, while such priors are also a major source of hallucinations. This suggests that enhancing hallucinations can expose hidden backdoor-related priors. Specifically, backdoored VLMs tend to hallucinate backdoor target tokens more frequently across diverse inputs, while the hallucinated tokens of clean VLMs are largely random and exhibit no stable target-specific bias. Based on this observation, we propose HalluProbe, which enhances hallucinations during output decoding and measures the distributional consistency of hallucinated tokens to identify backdoored VLMs. We evaluate HalluProbe against six representative VLM backdoor attacks across three models. Experimental results demonstrate that HalluProbe can effectively detect backdoored VLMs with an accuracy exceeding 90%, outperforming existing backdoor detection methods.
cs.CR · cs.AI