Latent Riemannian Flow Matching for Geometry-Grounded 3D Foundation Models
Lisa Weijler, Irene Ballester, Guofeng Mei, Tolga Birdal, Pedro Hermosilla
原文摘要
Geometric foundation models such as the Visual Geometry Grounded Transformer (VGGT) provide strong 3D priors from unposed images, but operate deterministically and cannot generate plausible geometry beyond what the observations directly support. We bridge geometric foundation models and generative 3D modeling by performing flow matching directly in their latent space, leveraging learned 3D priors without committing to explicit representations such as Gaussians, meshes, or video-VAE latents. This requires respecting the latent geometry: LayerNorm places the decoder inputs on a product of zero-mean hyperspheres, and treating this space as unconstrained Euclidean substantially degrades flow matching. We therefore formulate Riemannian flow matching on this product manifold, conditioned on unposed context views and a target camera pose, keeping generated tokens within the geometry expected by the frozen decoder. Our main experiments use VGGT, while additional experiments with Depth Anything 3 show that the same formulation transfers to a different independently pretrained 3D backbone without modification. On RealEstate10K, ScanNet++, and ETH3D, our method achieves strong results against recent scene-generation baselines in both novel-view appearance and aggregated 3D geometry, establishing latent-space flow matching on geometric foundation models as a viable paradigm for 3D generation.
cs.CV