← 研踪 PaperTrailarXiv:2602.15400 · v2
One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation
首次提交 2026-02-17 15:13
版本更新 2026-11-07 19:52
雷达观测 2026-11-10 11:14
摘要
A navigation agent must understand high-level semantic instructions and reason about precise spatial structure. Multimodal Large Language Models (MLLMs), with their strong generalization, are a promising foundation for such agents. We propose GTA, a decoupled design that separates low-level spatial state estimation from high-level semantic planning. We introduce an explicit metric world representation that combines accumulated geometry with navigation history and exposes this information to the MLLM through a multimodal interface for decision-making. We further introduce counterfactual branch reasoning that checks anticipated outcomes against subsequent observations and revises navigation decisions when they disagree. The world representation grounds waypoint selection in reconstructed metric geometry. GTA achieves 48.8% Success Rate (SR) on the full R2R-CE val-unseen split and 46.2% on a 260-episode English RxR-CE val-unseen subset. Additional comparisons on shared R2R-CE episodes and mechanism ablations support the framework's effectiveness. We also demonstrate transfer to wheeled, quadrupedal, and aerial robots. Project page: https://zeruili22.github.io/GTA/
作者备注：CoRL 2026
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 19:52 · 最新观测版本
One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation
A navigation agent must understand high-level semantic instructions and reason about precise spatial structure. Multimodal Large Language Models (MLLMs), with their strong generalization, are a promising foundation for such agents. We propose GTA, a decoupled design that separates low-level spatial state estimation from high-level semantic planning. We introduce an explicit metric world representation that combines accumulated geometry with navigation history and exposes this information to the MLLM through a multimodal interface for decision-making. We further introduce counterfactual branch reasoning that checks anticipated outcomes against subsequent observations and revises navigation decisions when they disagree. The world representation grounds waypoint selection in reconstructed metric geometry. GTA achieves 48.8% Success Rate (SR) on the full R2R-CE val-unseen split and 46.2% on a 260-episode English RxR-CE val-unseen subset. Additional comparisons on shared R2R-CE episodes and mechanism ablations support the framework's effectiveness. We also demonstrate transfer to wheeled, quadrupedal, and aerial robots. Project page: https://zeruili22.github.io/GTA/
查看此版本解读 →查看此版本原文 ↗