← 研踪 PaperTrailarXiv:2608.22294 · v3
Semantic Role-Structured World Model for Physical Interaction Planning
首次提交 2026-08-23 16:37
版本更新 2026-11-19 17:58
雷达观测 2026-11-20 19:57
摘要
World models for robotic manipulation must capture how actions transform a scene in ways that are meaningful for task execution. Existing object-centric approaches preserve the transient identity of entities, while leaving the evolving task structure that gives those dynamics meaning largely implicit. We introduce SemRole, a world model that represents scene evolution through semantic roles: a task-conditioned structure explicitly encoding the functional status of scene entities, their task-relevant relations, and the current phase of execution. Conditioned on language, visual observations, and proprioception, SemRole uses this structured state to govern action generation and action-conditioned dynamics, predicting interaction events, relation satisfaction, and phase progression. At inference time, these predictions guide phase-aware action selection and local repair of invalid rollouts. We instantiate SemRole on both lightweight LeWM and pretrained Cosmos backbones, improving closed-loop success from 45.5% to 83.0% on LIBERO, 42.9% to 55.2% on RoboCasa, and 55.0% to 81.0% on real-robot Galbot transfer. Across backbones and domains, explicitly modeling task-semantic structure yields substantially stronger world-model-based manipulation.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-19 17:58 · 最新观测版本
Semantic Role-Structured World Model for Physical Interaction Planning
World models for robotic manipulation must capture how actions transform a scene in ways that are meaningful for task execution. Existing object-centric approaches preserve the transient identity of entities, while leaving the evolving task structure that gives those dynamics meaning largely implicit. We introduce SemRole, a world model that represents scene evolution through semantic roles: a task-conditioned structure explicitly encoding the functional status of scene entities, their task-relevant relations, and the current phase of execution. Conditioned on language, visual observations, and proprioception, SemRole uses this structured state to govern action generation and action-conditioned dynamics, predicting interaction events, relation satisfaction, and phase progression. At inference time, these predictions guide phase-aware action selection and local repair of invalid rollouts. We instantiate SemRole on both lightweight LeWM and pretrained Cosmos backbones, improving closed-loop success from 45.5% to 83.0% on LIBERO, 42.9% to 55.2% on RoboCasa, and 55.0% to 81.0% on real-robot Galbot transfer. Across backbones and domains, explicitly modeling task-semantic structure yields substantially stronger world-model-based manipulation.
查看此版本解读 →查看此版本原文 ↗