← 研踪 PaperTrailarXiv:2506.23532 · v2
GViT: Representing Images as Gaussians for Visual Recognition
首次提交 2025-06-30 13:44
版本更新 2026-10-28 02:50
雷达观测 2026-10-30 10:23
摘要
We introduce GViT, a supervised classification framework that uses patch-conditioned 2D Gaussian primitives as input tokens for ViT backbones. A ViT-S-sized encoder predicts each primitive's position, scale, orientation, color, and opacity in one forward pass. The same parameters are constrained to reconstruct the image through a differentiable renderer, as well as serve as classifier inputs. During training, classifier-gradient guidance shapes primitive geometry and appearance under an image reconstruction constraint. On ImageNet-1K, GViT-B achieves 76.9% top-1 accuracy, compared with 79.8% for ViT-B/16 trained with the same recipe. Gaussian tokens also provide explicit spatial support for explanation maps. Grad-CAM over Gaussians improves on patch-based Grad-CAM in segmentation and token-deletion probes, while other probe comparisons are comparable to a ViT. A covariance-based foreground diagnostic on the PASCAL VOC segmentation task further measures the geometry learned through supervised guidance. These results establish the feasibility of recognition from renderable Gaussian parameters and characterize how task supervision shapes this inspectable representation.
作者备注：- added new explainability experiments - more clarification about our method - more comparisons with other methods
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-28 02:50 · 最新观测版本
GViT: Representing Images as Gaussians for Visual Recognition
We introduce GViT, a supervised classification framework that uses patch-conditioned 2D Gaussian primitives as input tokens for ViT backbones. A ViT-S-sized encoder predicts each primitive's position, scale, orientation, color, and opacity in one forward pass. The same parameters are constrained to reconstruct the image through a differentiable renderer, as well as serve as classifier inputs. During training, classifier-gradient guidance shapes primitive geometry and appearance under an image reconstruction constraint. On ImageNet-1K, GViT-B achieves 76.9% top-1 accuracy, compared with 79.8% for ViT-B/16 trained with the same recipe. Gaussian tokens also provide explicit spatial support for explanation maps. Grad-CAM over Gaussians improves on patch-based Grad-CAM in segmentation and token-deletion probes, while other probe comparisons are comparable to a ViT. A covariance-based foreground diagnostic on the PASCAL VOC segmentation task further measures the geometry learned through supervised guidance. These results establish the feasibility of recognition from renderable Gaussian parameters and characterize how task supervision shapes this inspectable representation.
查看此版本解读 →查看此版本原文 ↗