ElasticDino:使用弹性变形生成稠密语义特征
ElasticDino: Dense Semantic Features Using Elastic Deformations
Ulysse Mizrahi, Sean Zaretzky, Dov Danon, Amit H. Bermano
AI 速读 基于摘要
ElasticDino 是一种轻量适配器,通过空间弹性变形将 ViT 特征对齐到输入图像结构,高效生成高分辨率、像素对齐的稠密语义特征图。
cs.CV · cs.AI
查看原文摘要
Vision transformers (ViTs) serve as the backbone for countless vision tasks, including dense prediction tasks, such as segmentation and depth estimation, even though the quadratic complexity of self-attention severely limits feature resolution and induces spatial artifacts. In this work, we introduce ElasticDino, a lightweight adapter which produces high-resolution feature maps, adhering to the fine details of the input image. As a key insight, we observe the consistency of artifacts across channels, and simultaneously resolve them to match the input image structure. Using a slim architecture based on spatial deformations, ElasticDino produces pixel-aligned, high-resolution feature maps efficiently. Leveraging deformations allows strictly preserving sharp semantics, and thereby avoiding the smoothness biases inherent to alternative approaches. Due to the absence of ground truth, we employ pixel-level semantic segmentation and depth estimation with linear heads to predict a set of downstream dense tasks, guiding the deformation process without leaking task-specific semantics into the model. We demonstrate superior accuracy over current methods on a range of popular dense task benchmarks, with better computational performance.