← 研踪 PaperTrailarXiv:2607.05133 · v2
UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
首次提交 2026-07-06 22:18
版本更新 2026-11-16 16:43
雷达观测 2026-11-17 17:11
摘要
World Action Models (WAMs) can improve action generalization in autonomous driving by learning scene dynamics through dense future-video supervision. However, deploying such models often requires costly future-video generation, while architectures that bypass video rollout typically separate video and action parameters or alter their learned dependencies at inference. We introduce UNIVERSE, a unified video-action model built on a single mask-modulated Diffusion Transformer (DiT) that combines shared-parameter learning with modality-independent inference. Its Modality-Decoupling Visibility Mask allows future video and trajectory tokens to attend to a common historical context while blocking mutual attention between the two future modalities. As a result, future-video supervision directly updates the parameters used for trajectory denoising, yet video tokens can be omitted at test time without changing the trajectory branch's training-time attention dependencies. The same model supports trajectory-only, video-only, and joint video-action generation. Trajectory-only inference achieves a 4.3x speedup over coupled video-action rollout with comparable planning accuracy. UNIVERSE achieves 91.0 PDMS on NAVSIM, compared with 89.6 for a Two-DiT variant, and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without target-domain fine-tuning. Controlled ablations further support the complementary benefits of shared parameters, dense video co-training, and modality-decoupled visibility.
作者备注:Accepted by NeurIPS 2026. Code: https://github.com/xiaomi-mlab/UNIVERSE
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-16 16:43 · 最新观测版本
UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation
World Action Models (WAMs) can improve action generalization in autonomous driving by learning scene dynamics through dense future-video supervision. However, deploying such models often requires costly future-video generation, while architectures that bypass video rollout typically separate video and action parameters or alter their learned dependencies at inference. We introduce UNIVERSE, a unified video-action model built on a single mask-modulated Diffusion Transformer (DiT) that combines shared-parameter learning with modality-independent inference. Its Modality-Decoupling Visibility Mask allows future video and trajectory tokens to attend to a common historical context while blocking mutual attention between the two future modalities. As a result, future-video supervision directly updates the parameters used for trajectory denoising, yet video tokens can be omitted at test time without changing the trajectory branch's training-time attention dependencies. The same model supports trajectory-only, video-only, and joint video-action generation. Trajectory-only inference achieves a 4.3x speedup over coupled video-action rollout with comparable planning accuracy. UNIVERSE achieves 91.0 PDMS on NAVSIM, compared with 89.6 for a Two-DiT variant, and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without target-domain fine-tuning. Controlled ablations further support the complementary benefits of shared parameters, dense video co-training, and modality-decoupled visibility.
查看此版本解读 →查看此版本原文 ↗