Depth perception is fundamental for autonomous navigation, industrial inspection, and mixed-reality systems. It often operates under strict constraints on power and form factor. In such cases, monocular depth estimation (MDE) is often used. Even as recent advances continue to improve monocular depth quality, conventional frame-based MDE systems remain challenged by fast motion and high-dynamic-range scenes. This work introduces eSharp, an MDE framework that addresses these problems using an event camera. eSharp repurposes pretrained diffusion priors for event-only MDE and produces accurate metric depth with unparalleled sharpness on real data. Real-world event datasets provide only sparse LiDAR depth maps for ground-truth supervision, making dense prediction challenging. We address this limitation through cross-modal distillation, using LiDAR-calibrated dense pseudo-labels generated from a spatially co-registered frame camera to train eSharp. Experiments show that eSharp outperforms prior event-to-depth baselines in both metric-depth accuracy and spatial fidelity, establishing diffusion models as a promising foundation for event-driven 3D perception.
Jefferson Hernandez, Ruozhen He, Guha Balakrishnan, Alexander C. Berg, Vicente Ordonez
原文摘要
We introduce GViT, a supervised classification framework that uses patch-conditioned 2D Gaussian primitives as input tokens for ViT backbones. A ViT-S-sized encoder predicts each primitive's position, scale, orientation, color, and opacity in one forward pass. The same parameters are constrained to reconstruct the image through a differentiable renderer, as well as serve as classifier inputs. During training, classifier-gradient guidance shapes primitive geometry and appearance under an image reconstruction constraint. On ImageNet-1K, GViT-B achieves 76.9% top-1 accuracy, compared with 79.8% for ViT-B/16 trained with the same recipe. Gaussian tokens also provide explicit spatial support for explanation maps. Grad-CAM over Gaussians improves on patch-based Grad-CAM in segmentation and token-deletion probes, while other probe comparisons are comparable to a ViT. A covariance-based foreground diagnostic on the PASCAL VOC segmentation task further measures the geometry learned through supervised guidance. These results establish the feasibility of recognition from renderable Gaussian parameters and characterize how task supervision shapes this inspectable representation.