← 研踪 PaperTrailarXiv:2605.13450 · v2
Do Semantic Distance Tests Actually Predict Creativity in LLMs? Evidence for Jagged Creativity
首次提交 2026-05-13 20:46
版本更新 2026-11-07 11:28
雷达观测 2026-11-10 10:11
摘要
Measuring the creativity of large language models (LLMs) is essential for designing methods that can improve creativity and for enhancing our scientific understanding of this ability. To accomplish this, it has become common in recent years to administer semantic distance tests of human creativity to LLMs. Although these tests provide a convenient and fully automated way to score aspects of "creativity," their validity as measures of machine creativity has not been established. To address this problem, we conduct the first large-scale, systematic study assessing the effectiveness of semantic distance tests for predicting the creative achievement of LLMs across four target constructs: (a) creative writing, (b) output diversity on open-ended prompts, (c) combinatorial creativity, and (d) scientific ideation. We score each test on two criteria (validity and specificity), and derive a theoretical limit for the maximum attainable specificity and validity a test can achieve. We find that: (1) No test is a significant predictor of output diversity, suggesting that point creativity, the ability to produce a single response connecting semantically distant concepts, and distributional creativity, the ability to produce distinct responses across repeated samples, may dissociate in current LLMs--a pattern we call jagged creativity. (2) The Remote Associates Test is effectively a proxy for general capabilities rather than a specific predictor of creativity in LLMs. (3) Lastly, the Divergent Association Task is the best predictor of creative writing, but no single test predicts all constructs well. In summary, our results prescribe caution when applying psychometric tests and tasks of creativity validated among human populations to LLMs.
作者备注:Accepted at 40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026)
观测到的版本
相邻记录可能跳过中间版本;首次发现 v2 及以上也会标为版本更新。
v2 · 2026-11-07 11:28 · 最新观测版本
Do Semantic Distance Tests Actually Predict Creativity in LLMs? Evidence for Jagged Creativity
Measuring the creativity of large language models (LLMs) is essential for designing methods that can improve creativity and for enhancing our scientific understanding of this ability. To accomplish this, it has become common in recent years to administer semantic distance tests of human creativity to LLMs. Although these tests provide a convenient and fully automated way to score aspects of "creativity," their validity as measures of machine creativity has not been established. To address this problem, we conduct the first large-scale, systematic study assessing the effectiveness of semantic distance tests for predicting the creative achievement of LLMs across four target constructs: (a) creative writing, (b) output diversity on open-ended prompts, (c) combinatorial creativity, and (d) scientific ideation. We score each test on two criteria (validity and specificity), and derive a theoretical limit for the maximum attainable specificity and validity a test can achieve. We find that: (1) No test is a significant predictor of output diversity, suggesting that point creativity, the ability to produce a single response connecting semantically distant concepts, and distributional creativity, the ability to produce distinct responses across repeated samples, may dissociate in current LLMs--a pattern we call jagged creativity. (2) The Remote Associates Test is effectively a proxy for general capabilities rather than a specific predictor of creativity in LLMs. (3) Lastly, the Divergent Association Task is the best predictor of creative writing, but no single test predicts all constructs well. In summary, our results prescribe caution when applying psychometric tests and tasks of creativity validated among human populations to LLMs.
查看此版本解读 →查看此版本原文 ↗