Vision Transformers (ViTs) exhibit robust performance across diverse visual scenarios. However, their efficiency is constrained by excessive token counts. Token merging offers a viable solution for achieving efficient ViTs. Existing methods merge tokens based solely on specific characteristics within the attention mechanism, which changes significantly across layers. In this paper, we propose a novel training-free SAliency-Driven Token Merging (SAD-TM) approach by leveraging not only the semantic relevance in the attention space but also the latent visual saliency of input patches. Our SAD-TM is inspired by the discovery that saliency-based statistics can directly capture the causal relationship between model input and output, regardless of the layers. Based on the observation, we develop a method that is mathematically formulated to merge tokens with high saliency outliers. The principle behind our merging is that tokens with high saliency outliers usually imply inconsistencies with the global gradient direction, and thus can be merged safely. Besides, our systematic analysis indicates that class attention shows considerable variation across early blocks, so a deferred merging strategy is introduced to optimize the selection of merging rates. In a training-free manner, SAD-TM demonstrates superior performance across various ViT architectures. Especially, with a FLOPs compression of 23.08% on DeiT-Tiny, SAD-TM achieves a Top-1 Accuracy comparable to the pretrained baseline on ImageNet dataset. The code will be available soon.
尚无 AI 解读，展示原文摘要
cs.CV
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition · 2026 · 正式发表
Visual Autoregressive (VAR) models have demonstrated competitive performance with diffusion models in image generation by adopting a "next-scale" prediction paradigm that significantly reduces inference steps. However, VAR's progressive multi-scale generation leads to severe memory overhead due to KV cache accumulation across all scales, limiting practical deployment. Existing solutions either require training and deploying multiple specialized models or sacrifice generation quality.We observe a critical scale-depth asymmetric dependency in VAR: small scales (low-resolution tokens) are highly sensitive to network depth and require deep layers to capture global semantic information, while large scales (high-resolution tokens) exhibit remarkable robustness to depth reduction.Motivated by this insight, we propose VARiant, a unified supernet framework that enables dynamic depth adjustment within a single model through parameter sharing. Our approach employs an even-spacing layer selection strategy to construct quality-preserving subnetworks, and introduces a dynamic-ratio progressive training strategy that gradually transitions from joint optimization (full network to subnetwork ratio 2:8) to subnetwork optimization (ratio 10:0), effectively resolving the inherent optimization conflicts between full network and subnetworks in supernet training.Extensive experiments on ImageNet demonstrate that our method achieves Pareto-optimal trade-offs between generation quality and inference efficiency: by using full depth (30 layers) for the first 7 scales and a 16-layer subnetwork (50% depth) for subsequent scales, we obtain 50% cache reduction and 1.8x inference speedup with minimal quality loss (FID increases by only 0.3).Unlike approaches requiring deployment of multiple models, our single-model solution eliminates deployment complexity, supports zero-cost runtime depth switching, and seamlessly integrates into standard transformer inference frameworks, making it highly practical for resource-constrained scenarios.
尚无 AI 解读，展示原文摘要
cs.CV
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition · 2026 · 正式发表