← 研踪 PaperTrailarXiv:2611.25835 · v1
PSM-WSM: Predictively Sufficient Multimodal World Social Modeling
首次提交 2026-11-19 17:51
版本更新 2026-11-19 17:51
雷达观测 2026-11-20 19:57
摘要
Learning latent states that support future evolution remains an open problem for multimodal interactive systems. Existing approaches typically optimize representations for current semantic understanding or direct prediction, without explicitly defining what information a latent state should preserve to remain predictive of future dynamics. In this work, we investigate this problem through the principle of predictive sufficiency, which defines a predictive state as a latent variable that retains future-relevant information from historical multimodal observations while reducing redundant dependence on the observed history. Based on this principle, we propose Predictively Sufficient Multimodal Social World Modeling (PSM-WM), a framework that jointly learns predictive latent states and their stochastic temporal evolution through probabilistic state inference, event-conditioned transitions, and multi-step consistency optimization. Beyond forecasting performance, we introduce state-centric evaluations to measure whether learned representations satisfy predictive state properties, including predictive closure, residual history gain, and future semantic alignment. Experiments on Hi-EF, IEMOCAP, MELD, and Ego4D show that PSM-WM improves future multimodal interaction modeling and learns latent states with stronger predictive properties and robustness. These results indicate that predictive sufficiency provides an effective principle for constructing future-oriented multimodal states and highlight the importance of state construction in multimodal world modeling.
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-19 17:51 · 最新观测版本
PSM-WSM: Predictively Sufficient Multimodal World Social Modeling
Learning latent states that support future evolution remains an open problem for multimodal interactive systems. Existing approaches typically optimize representations for current semantic understanding or direct prediction, without explicitly defining what information a latent state should preserve to remain predictive of future dynamics. In this work, we investigate this problem through the principle of predictive sufficiency, which defines a predictive state as a latent variable that retains future-relevant information from historical multimodal observations while reducing redundant dependence on the observed history. Based on this principle, we propose Predictively Sufficient Multimodal Social World Modeling (PSM-WM), a framework that jointly learns predictive latent states and their stochastic temporal evolution through probabilistic state inference, event-conditioned transitions, and multi-step consistency optimization. Beyond forecasting performance, we introduce state-centric evaluations to measure whether learned representations satisfy predictive state properties, including predictive closure, residual history gain, and future semantic alignment. Experiments on Hi-EF, IEMOCAP, MELD, and Ego4D show that PSM-WM improves future multimodal interaction modeling and learns latent states with stronger predictive properties and robustness. These results indicate that predictive sufficiency provides an effective principle for constructing future-oriented multimodal states and highlight the importance of state construction in multimodal world modeling.
查看此版本解读 →查看此版本原文 ↗