跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2611.23770 · v1

DetectViT: Test-time Backdoor Detection for Vision Transformers via Inter-Head Attention Discrepancy

首次提交 2026-11-18 11:26
版本更新 2026-11-18 11:26
雷达观测 2026-11-19 19:08
arXiv ↗PDF ↗

摘要

Vision Transformers (ViTs) now underpin the visual encoders of most multimodal models, but this success comes at a cost: relying on third-party pretrained checkpoints leaves these systems open to backdoor attacks. Existing test-time detection methods rely on either the predicted label or intermediate embeddings, limiting their applicability across diverse tasks and incurring substantial computational overhead. In this work, we study how backdoor triggers affect the multi-head attention mechanism of ViTs and identify two distinct anomalous patterns: complete attention hijacking\textit{complete attention hijacking}, causing the attention to exhibit abnormally high inter-head spatial overlap, and partial attention hijacking\textit{partial attention hijacking}, causing an abnormally large variance in the concentration of the attention distribution. Motivated by these findings, we propose DetectViT\textbf{DetectViT}, a test-time backdoor detection method that requires neither training data, model outputs, nor any prior knowledge of the trigger. DetectViT quantifies the two hijacking phenomena via an inter-head consistency score and an inter-head entropy variance score, with thresholds estimated from a small out-of-distribution (OOD) calibration set drawn from an arbitrary source. Across four representative attacks, diverse architectures (DeiT, CLIP, and LLaVA), and downstream tasks (image classification and captioning), DetectViT substantially outperforms all baselines with as few as 64--128 OOD calibration samples, achieving a true positive rate of 100%100\% and a false positive rate of 0.17%0.17\% on BadCLIP. Notably, it introduces negligible additional overhead by relying solely on attention weights already computed during inference. Code is available at: https://github.com/siquanhuang/DetectViT.

作者备注:Accepted to NeurIPS 2026

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

目前只收录了一个版本。收录下一版后,可以对照标题、作者、分类和摘要。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v1 · 2026-11-18 11:26 · 最新观测版本

DetectViT: Test-time Backdoor Detection for Vision Transformers via Inter-Head Attention Discrepancy

Vision Transformers (ViTs) now underpin the visual encoders of most multimodal models, but this success comes at a cost: relying on third-party pretrained checkpoints leaves these systems open to backdoor attacks. Existing test-time detection methods rely on either the predicted label or intermediate embeddings, limiting their applicability across diverse tasks and incurring substantial computational overhead. In this work, we study how backdoor triggers affect the multi-head attention mechanism of ViTs and identify two distinct anomalous patterns: complete attention hijacking\textit{complete attention hijacking}, causing the attention to exhibit abnormally high inter-head spatial overlap, and partial attention hijacking\textit{partial attention hijacking}, causing an abnormally large variance in the concentration of the attention distribution. Motivated by these findings, we propose DetectViT\textbf{DetectViT}, a test-time backdoor detection method that requires neither training data, model outputs, nor any prior knowledge of the trigger. DetectViT quantifies the two hijacking phenomena via an inter-head consistency score and an inter-head entropy variance score, with thresholds estimated from a small out-of-distribution (OOD) calibration set drawn from an arbitrary source. Across four representative attacks, diverse architectures (DeiT, CLIP, and LLaVA), and downstream tasks (image classification and captioning), DetectViT substantially outperforms all baselines with as few as 64--128 OOD calibration samples, achieving a true positive rate of 100%100\% and a false positive rate of 0.17%0.17\% on BadCLIP. Notably, it introduces negligible additional overhead by relying solely on attention weights already computed during inference. Code is available at: https://github.com/siquanhuang/DetectViT.

查看此版本解读 →查看此版本原文 ↗