← 研踪 PaperTrailarXiv:2611.25435 · v1
VisualJevBench: A Benchmark and Black-Box Attacks for Visual Typed Decision Models
首次提交 2026-11-19 11:16
版本更新 2026-11-19 11:16
雷达观测 2026-11-20 10:50
摘要
Typed decision models (TDMs) return structured decisions instead of generating free-text responses, enabling fast and cheap decisions in real-world applications. Recent support for image inputs extends Jev, a representative TDM, to visual decision-making. We refer to this setting as Visual Jev. However, red teaming of Visual Jev leaves two gaps: a systematic evaluation of direct image manipulations is lacking, and universal visual attacks remain underexplored. To address these gaps, we introduce VisualJevBench, a benchmark for evaluating the adversarial robustness of Visual Jev. It comprises 800 images and 3,200 questions across eight scenarios and four visual attack surfaces: typographic manipulation, visual evidence spoofing, target selection interference, and geometric transformation. We also introduce Adaptive Black-Box Parameter Search (ABPS), a lightweight framework that uses typed-output feedback to find effective image manipulations. ABPS supports both image-specific attacks and universal attacks that transfer to unseen images and tasks in new scenarios without further optimization. Extensive experiments on jev-1.14.0 show that image-specific attacks achieve an attack success rate (ASR) of up to 69.6%, while universal attacks achieve an average ASR of 33.15% on unseen images. Geometric transformations achieve an average ASR of 40.09% in cross-scenario transfer.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-19 11:16 · 最新观测版本
VisualJevBench: A Benchmark and Black-Box Attacks for Visual Typed Decision Models
Typed decision models (TDMs) return structured decisions instead of generating free-text responses, enabling fast and cheap decisions in real-world applications. Recent support for image inputs extends Jev, a representative TDM, to visual decision-making. We refer to this setting as Visual Jev. However, red teaming of Visual Jev leaves two gaps: a systematic evaluation of direct image manipulations is lacking, and universal visual attacks remain underexplored. To address these gaps, we introduce VisualJevBench, a benchmark for evaluating the adversarial robustness of Visual Jev. It comprises 800 images and 3,200 questions across eight scenarios and four visual attack surfaces: typographic manipulation, visual evidence spoofing, target selection interference, and geometric transformation. We also introduce Adaptive Black-Box Parameter Search (ABPS), a lightweight framework that uses typed-output feedback to find effective image manipulations. ABPS supports both image-specific attacks and universal attacks that transfer to unseen images and tasks in new scenarios without further optimization. Extensive experiments on jev-1.14.0 show that image-specific attacks achieve an attack success rate (ASR) of up to 69.6%, while universal attacks achieve an average ASR of 33.15% on unseen images. Geometric transformations achieve an average ASR of 40.09% in cross-scenario transfer.
查看此版本解读 →查看此版本原文 ↗