← 研踪 PaperTrailarXiv:2607.18642 · v3
Spaghetti Architect: A Controlled Code Benchmark Generator, and What Its Own Comments Do to Model Scores
首次提交 2026-07-21 10:23
版本更新 2026-11-07 17:18
雷达观测 2026-11-10 11:14
摘要
Code corpora mined from the wild cannot change one thing at a time: a program's behaviour, messiness and size vary together, and nothing marks the correct answer. We present Spaghetti Architect, a generator that turns a small JSON description of a program into deliberately messy programs in five languages. It compiles and runs every program, checks the output against a reference computed from the description, and labels each program by problem size and by messiness. This control lets us change one property at a time and still know the correct answer. The generator writes comments into its own output, so we first re-render identical code without them. Removing the comments lowers the refactoring score of all four models we tested, and lowers the weakest model's about ten times as much as the strongest model's; a pre-registered replication confirms the difference at a smaller magnitude. The comments also hide an effect: messiness leaves scores flat while they are present and lowers them on both tasks once they are gone. We therefore recommend evaluating on the comment-free rendering. Changing only problem size drives direct-answer aggregation accuracy to zero for every model, and changing only literal values moves scores by less than 0.012. The generator, the dataset and the evaluation records are open source.
作者备注：Code: https://github.com/KurathSec/Spaghetti-Architect (artifact archived at doi:10.5281/zenodo.21033174)
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v3 · 2026-11-07 17:18 · 最新观测版本
Spaghetti Architect: A Controlled Code Benchmark Generator, and What Its Own Comments Do to Model Scores
Code corpora mined from the wild cannot change one thing at a time: a program's behaviour, messiness and size vary together, and nothing marks the correct answer. We present Spaghetti Architect, a generator that turns a small JSON description of a program into deliberately messy programs in five languages. It compiles and runs every program, checks the output against a reference computed from the description, and labels each program by problem size and by messiness. This control lets us change one property at a time and still know the correct answer. The generator writes comments into its own output, so we first re-render identical code without them. Removing the comments lowers the refactoring score of all four models we tested, and lowers the weakest model's about ten times as much as the strongest model's; a pre-registered replication confirms the difference at a smaller magnitude. The comments also hide an effect: messiness leaves scores flat while they are present and lowers them on both tasks once they are gone. We therefore recommend evaluating on the comment-free rendering. Changing only problem size drives direct-answer aggregation accuracy to zero for every model, and changing only literal values moves scores by less than 0.012. The generator, the dataset and the evaluation records are open source.
查看此版本解读 →查看此版本原文 ↗