← 研踪 PaperTrailarXiv:2611.15575 · v1
Does Confusing Code Confuse LLMs? A Pilot Study on C++
首次提交 2026-11-11 22:59
版本更新 2026-11-11 22:59
雷达观测 2026-11-13 14:16
摘要
Large language models (LLMs) are increasingly deployed for software maintenance tasks that depend heavily on code comprehension. While atoms of confusion, minimal code patterns empirically shown to impair human program comprehension, are well studied, it remains unclear whether these same patterns affect LLMs, and whether their impact varies across maintenance tasks. We present a pilot study evaluating the impact of atoms of confusion on LLM performance on C++ code drawn from three production libraries, using ChatGPT and GitHub Copilot on three software development tasks: atom injection and removal, output prediction, and code explanation. Our results reveal three preliminary findings: (i) the impact of atoms on LLMs is strongly task-dependent; (ii) atoms requiring reasoning about surrounding code are the most resistant to transformation, while localized syntactic atoms are handled reliably; and (iii) LLMs remove atoms of confusion more reliably than they inject them, suggesting that removal succeeds through pattern matching while injection requires deeper comprehension.
观测到的版本
相邻记录可能跳过中间版本；首次发现 v2 及以上也会标为版本更新。
v1 · 2026-11-11 22:59 · 最新观测版本
Does Confusing Code Confuse LLMs? A Pilot Study on C++
Large language models (LLMs) are increasingly deployed for software maintenance tasks that depend heavily on code comprehension. While atoms of confusion, minimal code patterns empirically shown to impair human program comprehension, are well studied, it remains unclear whether these same patterns affect LLMs, and whether their impact varies across maintenance tasks. We present a pilot study evaluating the impact of atoms of confusion on LLM performance on C++ code drawn from three production libraries, using ChatGPT and GitHub Copilot on three software development tasks: atom injection and removal, output prediction, and code explanation. Our results reveal three preliminary findings: (i) the impact of atoms on LLMs is strongly task-dependent; (ii) atoms requiring reasoning about surrounding code are the most resistant to transformation, while localized syntactic atoms are handled reliably; and (iii) LLMs remove atoms of confusion more reliably than they inject them, suggesting that removal succeeds through pattern matching while injection requires deeper comprehension.
查看此版本解读 →查看此版本原文 ↗