跳到正文
研踪PaperTrail
← 研踪 PaperTrail

arXiv:2607.05133 · v2

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

首次提交 2026-07-06 22:18
版本更新 2026-11-16 16:43
雷达观测 2026-11-17 17:11
arXiv ↗PDF ↗

摘要

World Action Models (WAMs) can improve action generalization in autonomous driving by learning scene dynamics through dense future-video supervision. However, deploying such models often requires costly future-video generation, while architectures that bypass video rollout typically separate video and action parameters or alter their learned dependencies at inference. We introduce UNIVERSE, a unified video-action model built on a single mask-modulated Diffusion Transformer (DiT) that combines shared-parameter learning with modality-independent inference. Its Modality-Decoupling Visibility Mask allows future video and trajectory tokens to attend to a common historical context while blocking mutual attention between the two future modalities. As a result, future-video supervision directly updates the parameters used for trajectory denoising, yet video tokens can be omitted at test time without changing the trajectory branch's training-time attention dependencies. The same model supports trajectory-only, video-only, and joint video-action generation. Trajectory-only inference achieves a 4.3x speedup over coupled video-action rollout with comparable planning accuracy. UNIVERSE achieves 91.0 PDMS on NAVSIM, compared with 89.6 for a Two-DiT variant, and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without target-domain fine-tuning. Controlled ablations further support the complementary benefits of shared parameters, dense video co-training, and modality-decoupled visibility.

作者备注:Accepted by NeurIPS 2026. Code: https://github.com/xiaomi-mlab/UNIVERSE

AI 论文解读

基于摘要 · 非全文审阅

此版本尚未生成 AI 解读。解读由后台任务生成并按论文版本保存,登录后共享已有结果。

版本差异

本站首次收录为 v2,暂无更早版本可对照。

观测到的版本

相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。

v2 · 2026-11-16 16:43 · 最新观测版本

UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

World Action Models (WAMs) can improve action generalization in autonomous driving by learning scene dynamics through dense future-video supervision. However, deploying such models often requires costly future-video generation, while architectures that bypass video rollout typically separate video and action parameters or alter their learned dependencies at inference. We introduce UNIVERSE, a unified video-action model built on a single mask-modulated Diffusion Transformer (DiT) that combines shared-parameter learning with modality-independent inference. Its Modality-Decoupling Visibility Mask allows future video and trajectory tokens to attend to a common historical context while blocking mutual attention between the two future modalities. As a result, future-video supervision directly updates the parameters used for trajectory denoising, yet video tokens can be omitted at test time without changing the trajectory branch's training-time attention dependencies. The same model supports trajectory-only, video-only, and joint video-action generation. Trajectory-only inference achieves a 4.3x speedup over coupled video-action rollout with comparable planning accuracy. UNIVERSE achieves 91.0 PDMS on NAVSIM, compared with 89.6 for a Two-DiT variant, and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without target-domain fine-tuning. Controlled ablations further support the complementary benefits of shared parameters, dense video co-training, and modality-decoupled visibility.

查看此版本解读 →查看此版本原文 ↗