← 研踪 PaperTrailarXiv:2606.24548 · v4
Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning
首次提交 2026-06-23 21:15
版本更新 2026-11-10 12:10
雷达观测 2026-11-11 13:08
摘要
Text-to-image (T2I) generation models have achieved remarkable progress in producing visually realistic images from natural language prompts. Yet it remains unclear whether their success reflects genuine causal understanding or sophisticated pattern matching over visual-textual correlations. Inspired by Russell's inductivist turkey, we introduce Counterfactual-World (CF-World), a counterfactual benchmark designed to investigate whether text-to-image models can generate images under rules that systematically contradict real-world priors. CF-World organizes each scenario into three progressive levels: factual generation under ordinary world knowledge, explicit counterfactual generation with direct visual instructions, and implicit counterfactual generation requiring causal deduction from altered rules. We evaluate open-source and closed-source text-to-image models using CF-Eval, a scoring pipeline based on vision-language models. We report the scores at each prompt level as the primary results and use Prior Resistance Rate (PRR) and Reasoning Retention Rate (RRR) as auxiliary descriptive indicators. All evaluated models score lower on counterfactual prompts than on factual prompts. Diagnostic experiments reveal difficulties in following altered rules and recombining rarely co-occurring visual attributes. The gains from rephrasing object and attribute names are consistent with a role for learned visual priors in counterfactual generation. Our benchmark and evaluation code are publicly available at https://jylei16.github.io/CF-World.github.io/.
作者备注：10 pages, 7 figures. Project page: https://github.com/jylei16/CF-World.github.io
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v4 · 2026-11-10 12:10 · 最新观测版本
Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning
Text-to-image (T2I) generation models have achieved remarkable progress in producing visually realistic images from natural language prompts. Yet it remains unclear whether their success reflects genuine causal understanding or sophisticated pattern matching over visual-textual correlations. Inspired by Russell's inductivist turkey, we introduce Counterfactual-World (CF-World), a counterfactual benchmark designed to investigate whether text-to-image models can generate images under rules that systematically contradict real-world priors. CF-World organizes each scenario into three progressive levels: factual generation under ordinary world knowledge, explicit counterfactual generation with direct visual instructions, and implicit counterfactual generation requiring causal deduction from altered rules. We evaluate open-source and closed-source text-to-image models using CF-Eval, a scoring pipeline based on vision-language models. We report the scores at each prompt level as the primary results and use Prior Resistance Rate (PRR) and Reasoning Retention Rate (RRR) as auxiliary descriptive indicators. All evaluated models score lower on counterfactual prompts than on factual prompts. Diagnostic experiments reveal difficulties in following altered rules and recombining rarely co-occurring visual attributes. The gains from rephrasing object and attribute names are consistent with a role for learned visual priors in counterfactual generation. Our benchmark and evaluation code are publicly available at https://jylei16.github.io/CF-World.github.io/.
查看此版本解读 →查看此版本原文 ↗