Giulia Marchiori Pietrosanti, Giulio Rossolini, Giorgio Buttazzo
原文摘要
Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks, e.g., adversarial patches, while recent test-time defenses mitigate them by suppressing image tokens with abnormally high attention scores. These defenses exploit a strong coupling between attention and adversarial effectiveness: adversarial tokens often need to attract substantial attention to influence the prediction. We introduce adversarial decoys, image patches optimized to redirect attention, and therefore related defenses, towards selected target tokens. Rather than jointly optimizing misclassifications and defense evasion, our approach decouples the two objectives: the original adversarial region induces the incorrect prediction, while a separate decoy manipulates the attention ranking used by the defense. A layer-wise objective increases target-token attention and promotes these tokens above competing non-target ones. Since the decoy optimization is not tied to a specific attack formulation, the method can be easily integrated with any existing adversarial patch attack. Experiments on ImageNet across multiple ViT architectures and adversarial patch attacks show that decoys consistently redirect high attention scores away from the true adversarial region while preserving much of the attack effectiveness against attention-ranking defenses. These results reveal a fundamental limitation of using attention alone as an indicator of adversarial relevance.
Aleksandar Fontana Scuola Superiore Sant'Anna Pisa, Marco Simoni, Giulio Rossolini Scuola Superiore Sant'Anna Pisa, Paolo Mori Istituto di Informatica e Telematica, Andrea Saracino Scuola Superiore Sant'Anna Pisa
原文摘要
Group-based reinforcement learning methods, like Group Relative Policy Optimization (GRPO), are widely used nowadays to post-train large language models. Despite their empirical success, they exhibit structural mismatches between reward optimization and the underlying training objective. In this paper, we present a theoretical analysis of GRPO style methods by studying them within a unified surrogate formulation. This perspective reveals recurring properties that affect all the methods under analysis: (i) non-uniform group weighting induces systematic gradient biases on shared prefix tokens; (ii) interactions with the AdamW optimizer make training dynamics largely insensitive to reward scaling; and (iii) optimizer momentum can push policy updates beyond the intended clipping region under repeated optimization steps. We believe that these findings highlight fundamental limitations of current approaches and provide principled guidance for the design of future formulations.
cs.CL
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · 2026 · 正式发表