Yihan Zang, Da Li, Shinkyu Park, Ivan Viola, Dominik Engel
Existing referring segmentation methods on Gaussian scenes primarily target whole objects, relying on segmenter-generated candidates from which a vision-language model (VLM) selects. Such predetermined fixed candidates may omit fine-grained parts needed as targets or cues for identifying objects. We introduce SplatScout, a query-driven, closed-loop approach that places perception under VLM control. The model actively selects views, directs segmentation, and inspects rendered 3D results to refine its predictions. For part-level evaluation, we introduce PartRef-ScanNet++, comprising 261 manually verified queries across 12 ScanNet++ scenes with mesh-derived ground truth. On PartRef-ScanNet++, SplatScout achieves 0.538 mIoU, substantially outperforming existing methods. Ablations demonstrate the benefits of iterative refinement, while comparisons across VLM backbones highlight the importance of effective tool-feedback utilization. SplatScout also achieves higher scores on four existing referring benchmarks. Code and the PartRef-ScanNet++ benchmark are available at https://github.com/kaust-violab/SplatScout.
Da Li, Yihan Zang, Ivan Viola, Zeyu Wang, Dominik Engel
Achieving high-quality reconstruction and responsive camera tracking remains challenging for monocular RGB Gaussian splatting SLAM under a limited optimization budget. We present an online system that addresses this challenge by combining learned geometric priors with explicit geometric camera tracking, which can substantially outperform learned camera prediction in terms of tracking accuracy. A frozen streaming 3D foundation model serves as a virtual RGB-D sensor, providing depth estimation from monocular video. An explicit geometric solver then estimates camera motion through feature reprojection, and a streaming Gaussian mapper integrates the resulting observations into a dense scene representation. We further show that geometric tracking improves camera pose accuracy across representative streaming geometry backbones, demonstrating the broader applicability of our approach. Extensive experiments on indoor and outdoor datasets show that our method significantly improves reconstruction quality over existing approaches, achieving about 7 FPS on a single RTX 4090 and delivering 2-3 times the throughput of the evaluated prior SLAM systems.
Donggang Jia, Da Li, Zainab Alsuwaykit, Sebastian Hartwig, Dominik Engel, Timo Ropinski, Ivan Viola
Long-horizon object-centered tasks in interactive 3D environments involve heterogeneous events that unfold across time and space, including fine-grained manipulations, larger structural motions, and subtle state transitions. Effectively visualizing such processes is essential for helping users understand how actions lead to outcomes over time. However, existing camera control methods in 3D systems are commonly designed for static objects, and often fail to maintain the legibility of task-relevant components and their evolving relationships throughout long-horizon executions. We present VISTA, a differentiable viewpoint planning framework for long-horizon task visualization. We formulate camera control as a visualization policy that optimizes camera parameters under a unified rendering-based quality objective and we propose five quality metrics to satisfy our five design goals. To support both local part inspection and interaction-centric visualization centered on a primary object, our formulation handles targets defined either as object--part pairs or as relational tuples involving another interacting object. This allows the planner to reveal not only what part is important, but also how it functionally relates to surrounding structures during task execution. We demonstrate that the same planner supports two complementary use cases: presentation-driven narrative visualization for communicating how long-horizon tasks unfold to human viewers, and agent-driven active observation within an observe-act loop. Together, these results position differentiable viewpoint planning as a general mechanism for ``how-things-work'' visualization of long-horizon object-centered interactions in 3D environments.