- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether models which achieve high test scores on the Divergent Association Task (DAT) also tend to give a wide range of diverse answers when prompted repeatedly, or whether they tend to exhibit signs of output homogenization that have been observed among LLMs elsewhere on open-ended tasks.
Critical strengths
S1: Important research question: This paper addresses an important weakness regarding the Divergent Association Task (DAT), a vocabulary-space, divergent thinking test designed by Olson (2021) [1] to assess divergent creativity in humans. The DAT only reports item-level scores instead of cross-trial diversity, but output homogenization is an important problem among LLMs, known as the artificial hivemind effect [2].
[1] Naming Unrelated Words Predicts Creativity (https://www.pnas.org/doi/10.1073/pnas.2022340118)
[2] Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (https://arxiv.org/abs/2510.22954)
Critical weaknesses
W1: Inference settings: The paper mentions testing various temperature, top-p, and top-k values, but does not explain how scores are aggregated across each of these configurations. As a result, the standard deviations for each model are all quite large (2-5 points) (Table 1 of the Appendix), and the DAT scores for GPT-5.4 and GPT-5.4-Mini reported in Section 3 are far below small open-weight models like Llama3 and Mistral. These findings contradict what previous large-scale studies have reported: [1] reports that GPT-5.4 scores and GPT-5.4-Mini scores , [5] reports that mean GPT-4 scores already outperform mean human scores using the same dataset from Olson (2021), and [4] reports that both GPT-4 and GPT-3.5-Turbo already outperform mean human scores on that same human dataset. The temperature settings also seem excessively large here, as other studies examining LLM performance test temperature [1, 4, 6].
W2: Only uses the DAT: Using the DAT as the sole object of study in this paper limits its practical impact. As shown in [1], the DAT has limited external validity in predicting creative performance in LLMs for real-world tasks like scientific ideation, storytelling, and output diversity. Moreover, there are many other vocabulary-space, semantic distance tests that extend the DAT and should be studied here to make the work more comprehensive. A few examples include the Conditional DAT [2], Parallel Association Chains for Creativity Evaluation (PACE) [3], and the Divergent Remote Association Test (DRAT) [1]. Moreover, some of these (CDAT, DRAT) include a convergent thinking component that makes them more representative of real-world creative ideation, in which task-specific demands must be met in order to ensure artifacts are appropriate and useful [7].
[1] Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers (https://arxiv.org/abs/2605.13450)
[2] Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models (https://arxiv.org/abs/2601.20546)
[3] Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models (https://arxiv.org/abs/2510.12110)
[4] Probing the "Creativity" of Large Language Models: Can Models Produce Semantic Association? (https://arxiv.org/pdf/2310.11158)
[5] Divergent Creativity in Humans and Large Language Models (https://arxiv.org/abs/2405.13012)
[6] A large-scale comparison of divergent creativity in humans and large language models (https://www.nature.com/articles/s41562-025-02331-1)
[7] Maher, Mary Lou. "Evaluating creativity in humans, computers, and collectively intelligent systems." Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design. 2010.
Questions for authors
- Can you test the robustness of these findings to varying embedding models besides GloVe?
- In Table 1 of the appendix, why are the minimum scores for GPT-5.4 and GPT-5.4-mini so low? Also, why are the maximum scores for GPT-5.4 and GPT-5.4-mini so far below the maximum scores for the small open-weight models like Mistral, Phi4, and Llama3?
2 comments
Reviewer Response
Thank you for your rebuttal and for adding the CDAT. After reading the other reviews, I maintain my score.
Thank you for the thorough feedback, which helped us improve our paper, and for the list of additional references. Below we answer your specific comments and questions. See also our answers regarding common points, including adding the second task, in the general response.
Q1: Other embeddings.
The comparison with humans holds under all three embeddings; the ranking among the models changes (Tables 13 and 14). Word overlap, TTR and entropy use no embeddings and are therefore unaffected.
Q2, Q3, and W1: GPT scores compared with earlier studies and with open models.
We do not dispute the previous results; the difference comes from the instructions and the embedding. Like the human DAT, our prompts ban specialised vocabulary. The prompts of [4] and [1] do not, running essentially an easier task. These studies report the highest GPT scores; [5] keeps the specialized vocabulary restriction. With a minimal prompt that does not restrict the vocabulary, GPT-5.4 scores 91.91 and GPT-5.4-mini 86.48 (91.72 and 84.06 in [1]), yet GPT-5.4 keeps 18 times the human word overlap and uses "quasar" in 99 of 100 trials (Table 15). Under fastText and word2vec, all models score above the human mean and within two points of each other (Table 13). The GPT minima (67.0 and 66.9) are higher than those of the open models (56.6 to 63.6). The maxima are lower because the GPT models use common words: 0.3% and 0.0% of their tokens fall below Zipf 3, against 11.8% for Mistral and 21.6% for Llama3 (Tables 1 and 12).
W1: Aggregation.
Model values are means over conditions: 108 per open model, 72 for GPT-5.4 and 60 for GPT-5.4-mini (Section 2). The SDs in Table 1 are trial-level and include the spread between conditions (GPT-5.4: 72.3 to 83.1 across prompt and reasoning conditions; Table 24).
W1: Temperatures.
[1] samples at temperatures 1.0 to 2.0, which we include. Temperature 8, possible only for the local models, tests whether earlier attempts to gain diversity from temperature were limited by the range they tried and investigate how settings move one model's single-trial score and cross-trial diversity by taking an extreme value (Section 2). At temperature 1 alone the ordering of models is unchanged (Table 10).
W2. Only the DAT task in the original submission.
We agree that multiple tasks are more informative than one. The revision adds the Conditional DAT [2], compared against human responses to the same cues (Section 3.2). [1] found other tasks to be limited, including PACE, which mainly serves as a proxy for capability, not creativity. DRAT (Albinsson and Lief 2023), which, as far as we know, is not included in [1], is yet to be validated as a creativity test. LLMs’ novelty score on CDAT is near the human mean, yet two answers of one model to the same cue share 1.9 to 5.2 times as many words as two people's answers. Its appropriateness score covers the convergent component you emphasize. We also agree that the DAT is not a general creativity measure: while [1] finds it to be the best predictor of creative writing, no test predicts output diversity. Our cross-trial metrics complementing task scores aim to cover this gap.