← 研踪 PaperTrailarXiv:2605.20299 · v2
Mechanisms of Misgeneralization in Physical Sequence Modeling
首次提交 2026-05-19 20:34
版本更新 2026-11-07 02:25
雷达观测 2026-11-09 13:22
摘要
Generative sequence models are often trained to plan motion in physical domains, from robotics to mechanical simulations. When constructing a dataset to train such a model, engineers may curate demonstrations to specify how trajectories should be distributed over a physical quantity like travel distance or mechanical energy. For example, a roboticist building a maze navigation agent might choose demonstrations whose travel distances cover a fixed range uniformly, hoping to constrain the agent's expected power usage. We find that standard deep learning can violate this intent: each generated trajectory can seem plausible on its own, but the aggregate distribution over the physical quantity is wrong. We call this failure physical misgeneralization. In this work, we identify and formalize the problem, develop and validate an account of its mechanism, and use that account to compare and design mitigations. Using controlled synthetic tasks, we show that physical misgeneralization arises when local errors typical of the model class propagate through the physical measurement to shift the recovered distribution. We estimate these errors with a data deviation kernel, and we use it to predict which physical quantity values gain or lose mass under real diffusion sequence models trained on both toy and applied trajectory generation tasks. Finally, our mechanistic interpretation helps identify which mitigations are structurally promising. We first compare interventions that modify the dataset and generative interface, then show that the mechanistic understanding we develop throughout this paper suggests a novel way to construct data representations informed by the kernel. We show that our proposed mitigation, which transforms the coordinate system under which the generator learns and samples, effectively reduces quantity drift.
作者备注：40th Conference on Neural Information Processing Systems (NeurIPS 2026). kentonishi.com/physical-misgeneralization
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 02:25 · 最新观测版本
Mechanisms of Misgeneralization in Physical Sequence Modeling
Generative sequence models are often trained to plan motion in physical domains, from robotics to mechanical simulations. When constructing a dataset to train such a model, engineers may curate demonstrations to specify how trajectories should be distributed over a physical quantity like travel distance or mechanical energy. For example, a roboticist building a maze navigation agent might choose demonstrations whose travel distances cover a fixed range uniformly, hoping to constrain the agent's expected power usage. We find that standard deep learning can violate this intent: each generated trajectory can seem plausible on its own, but the aggregate distribution over the physical quantity is wrong. We call this failure physical misgeneralization. In this work, we identify and formalize the problem, develop and validate an account of its mechanism, and use that account to compare and design mitigations. Using controlled synthetic tasks, we show that physical misgeneralization arises when local errors typical of the model class propagate through the physical measurement to shift the recovered distribution. We estimate these errors with a data deviation kernel, and we use it to predict which physical quantity values gain or lose mass under real diffusion sequence models trained on both toy and applied trajectory generation tasks. Finally, our mechanistic interpretation helps identify which mitigations are structurally promising. We first compare interventions that modify the dataset and generative interface, then show that the mechanistic understanding we develop throughout this paper suggests a novel way to construct data representations informed by the kernel. We show that our proposed mitigation, which transforms the coordinate system under which the generator learns and samples, effectively reduces quantity drift.
查看此版本解读 →查看此版本原文 ↗