← 研踪 PaperTrailarXiv:2606.31689 · v2
ScratchWorld: Evaluating If World Models Compute Executable Consequences
首次提交 2026-06-30 22:02
版本更新 2026-10-28 08:51
雷达观测 2026-10-30 10:23
摘要
Evaluations of world models commonly score a predicted future state by its overlap with the ground truth. Yet in interactive worlds, this rewards copying: a model that merely preserves persistent state appears accurate without predicting any consequence of an action. We introduce ScratchWorld, an execution-grounded diagnostic benchmark that treats Scratch projects as executable worlds and derives every label from replay on a pinned Scratch VM, including next-state transitions, hidden variables, causal traces, and counterfactual outcomes. Its primary metric, value-aware changed-field F1 ($F_1^{\mathrm{VA}}$), credits a prediction only when it names a changed field and computes its executed value. Model-free anchors make the confound explicit: a no-change prior attains 91.4% full-state field accuracy on synthetic long-horizon items and 0% changed-field credit on non-empty one-step items, whereas VM replay attains 100% wherever the corresponding metric is defined. Across seven frontier LLMs prompted as world models, state-only $F_1^{\mathrm{VA}}$ reaches at most 30.0% on controlled synthetic mechanisms and declines steeply as projects approach real-world complexity. Revealing the full program raises every model's value-aware score by 22.3-51.7 points and substantially reorders the models, yet the best reaches only 55.4%, an execution gap that persists even with the rules visible. Auxiliary diagnostics complete the error profile: causal-event accuracy remains within 2.5 points of the 50% no-event baseline, with no correct exact chain; counterfactual prediction trades changed-outcome accuracy against invariance; hidden-state accuracy degrades roughly twice as fast as full-state overlap over long horizons. In a word, ScratchWorld separates what a model knows about the rules from what it can actually execute.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v2 · 2026-10-28 08:51 · 最新观测版本
ScratchWorld: Evaluating If World Models Compute Executable Consequences
Evaluations of world models commonly score a predicted future state by its overlap with the ground truth. Yet in interactive worlds, this rewards copying: a model that merely preserves persistent state appears accurate without predicting any consequence of an action. We introduce ScratchWorld, an execution-grounded diagnostic benchmark that treats Scratch projects as executable worlds and derives every label from replay on a pinned Scratch VM, including next-state transitions, hidden variables, causal traces, and counterfactual outcomes. Its primary metric, value-aware changed-field F1 ($F_1^{\mathrm{VA}}$), credits a prediction only when it names a changed field and computes its executed value. Model-free anchors make the confound explicit: a no-change prior attains 91.4% full-state field accuracy on synthetic long-horizon items and 0% changed-field credit on non-empty one-step items, whereas VM replay attains 100% wherever the corresponding metric is defined. Across seven frontier LLMs prompted as world models, state-only $F_1^{\mathrm{VA}}$ reaches at most 30.0% on controlled synthetic mechanisms and declines steeply as projects approach real-world complexity. Revealing the full program raises every model's value-aware score by 22.3-51.7 points and substantially reorders the models, yet the best reaches only 55.4%, an execution gap that persists even with the rules visible. Auxiliary diagnostics complete the error profile: causal-event accuracy remains within 2.5 points of the 50% no-event baseline, with no correct exact chain; counterfactual prediction trades changed-outcome accuracy against invariance; hidden-state accuracy degrades roughly twice as fast as full-state overlap over long horizons. In a word, ScratchWorld separates what a model knows about the rules from what it can actually execute.
查看此版本解读 →查看此版本原文 ↗