Cognitive Foundations for Reasoning and Their Manifestation in LLMs
Shuyue Stella Li, Priyanka Kargupta, Haocheng Wang, Jinu Lee, Dean Light, Shan Chen, Orevaoghene Ahia, Thomas L. Griffiths 等 12 位作者
Large language models (LLMs) solve complex problems yet fail on simpler variants, suggesting they achieve correct outputs through mechanisms fundamentally different from human reasoning. To understand this gap, we synthesize cognitive science research into a taxonomy of 28 cognitive elements spanning reasoning invariants, meta-cognitive controls, representations for organizing reasoning and knowledge, and transformation operations. We introduce a fine-grained evaluation framework and apply it to 192K traces from 18 models across text, vision, and audio, alongside 1,346 human think-aloud traces on a new, domain-agnostic dataset for human-model comparison. We find that models are able to deploy cognitive elements seen in human reasoning but differ in how they use them. Leveraging these patterns, we develop test-time reasoning guidance that automatically scaffolds successful reasoning topologies, improving the accuracy of the strongest models by 9.7 percentage points on ill-structured problems. Meta-analysis of 1.6K LLM reasoning papers reveals the research community concentrates on easily quantifiable elements but rarely targets self-awareness, whose use differs between human and model reasoning. By establishing a shared vocabulary between cognitive science and LLM research, our framework enables diagnosis of reasoning failures and principled development of models that reason through robust cognitive mechanisms rather than spurious shortcuts, while providing tools to test theories of human cognition at scale.
尚无 AI 解读，展示原文摘要