acceptodds
Under review as a conference paper at ICLR 2027

High Scores, Narrow Vocabularies: Measuring Cross-Trial Diversity in Large Language Models

Abstract

Large language models are increasingly used in creative industries as sources of novel ideas, treating output variation as evidence of genuine creative exploration. The Divergent Association Task (DAT), a validated creativity benchmark, has repeatedly produced at- or above-human level scores for LLMs, but a single-trial score cannot distinguish broad semantic exploration from repeated draws on a narrow, high-scoring vocabulary. We demonstrate this dissociation directly: LLMs can attain human-level or above-human DAT scores while showing substantially narrower cross-trial vocabularies than humans. We test five models, three open-source and two frontier, with 100 trials per condition under varied prompts, reasoning instructions, and decoding parameters, assessed using five metrics: DAT score, word overlap, vocabulary breadth, entropy, and semantic-region overlap. Prompt wording and reasoning instructions are the strongest predictors of DAT score, but they do not reliably increase lexical diversity. By contrast, widening the sampling pool reliably increases diversity without a corresponding gain in score for most models. Therefore, neither lever, within the standard decoding settings, closes the gap between benchmark performance and genuine exploration. Frontier models show the most extreme version of convergence, showing the highest cross-trial overlap and the narrowest vocabulary of any model tested. These findings show that human-like DAT performance does not imply human-like diversity. We propose multiple metrics of cross-trial diversity as a behavioral diagnostic of generative convergence that single-trial scores cannot reveal. As LLMs become routine in creative work, such convergence may homogenize otherwise independent outputs, warranting caution in treating the properties of individual outputs as evidence of genuine creative diversity.

Primary area:
generative models
Keywords:
Large language models, Divergent Association Task, divergent thinking, lexical diversity, semantic diversity, cross-trial diversity, creativity assessment, generative AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C6FKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether models which achieve high test scores on the Divergent Association Task (DAT) also tend to give a wide range of diverse answers when prompted repeatedly, or whether they tend to exhibit signs of output homogenization that have been observed among LLMs elsewhere on open-ended tasks.

Critical strengths

S1: Important research question: This paper addresses an important weakness regarding the Divergent Association Task (DAT), a vocabulary-space, divergent thinking test designed by Olson (2021) [1] to assess divergent creativity in humans. The DAT only reports item-level scores instead of cross-trial diversity, but output homogenization is an important problem among LLMs, known as the artificial hivemind effect [2].

[1] Naming Unrelated Words Predicts Creativity (https://www.pnas.org/doi/10.1073/pnas.2022340118)

[2] Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (https://arxiv.org/abs/2510.22954)

Critical weaknesses

W1: Inference settings: The paper mentions testing various temperature, top-p, and top-k values, but does not explain how scores are aggregated across each of these configurations. As a result, the standard deviations for each model are all quite large (2-5 points) (Table 1 of the Appendix), and the DAT scores for GPT-5.4 and GPT-5.4-Mini reported in Section 3 are far below small open-weight models like Llama3 and Mistral. These findings contradict what previous large-scale studies have reported: [1] reports that GPT-5.4 scores 91.72±0.1991.72 \pm 0.19 and GPT-5.4-Mini scores 84.06±0.3284.06 \pm 0.32, [5] reports that mean GPT-4 scores already outperform mean human scores using the same dataset from Olson (2021), and [4] reports that both GPT-4 and GPT-3.5-Turbo already outperform mean human scores on that same human dataset. The temperature 2,82, 8 settings also seem excessively large here, as other studies examining LLM performance test temperature T∈[0.5,1.5]T \in [0.5, 1.5] [1, 4, 6].

W2: Only uses the DAT: Using the DAT as the sole object of study in this paper limits its practical impact. As shown in [1], the DAT has limited external validity in predicting creative performance in LLMs for real-world tasks like scientific ideation, storytelling, and output diversity. Moreover, there are many other vocabulary-space, semantic distance tests that extend the DAT and should be studied here to make the work more comprehensive. A few examples include the Conditional DAT [2], Parallel Association Chains for Creativity Evaluation (PACE) [3], and the Divergent Remote Association Test (DRAT) [1]. Moreover, some of these (CDAT, DRAT) include a convergent thinking component that makes them more representative of real-world creative ideation, in which task-specific demands must be met in order to ensure artifacts are appropriate and useful [7].

[1] Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers (https://arxiv.org/abs/2605.13450)

[2] Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models (https://arxiv.org/abs/2601.20546)

[3] Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models (https://arxiv.org/abs/2510.12110)

[4] Probing the "Creativity" of Large Language Models: Can Models Produce Semantic Association? (https://arxiv.org/pdf/2310.11158)

[5] Divergent Creativity in Humans and Large Language Models (https://arxiv.org/abs/2405.13012)

[6] A large-scale comparison of divergent creativity in humans and large language models (https://www.nature.com/articles/s41562-025-02331-1)

[7] Maher, Mary Lou. "Evaluating creativity in humans, computers, and collectively intelligent systems." Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design. 2010.

Questions for authors
  • Can you test the robustness of these findings to varying embedding models besides GloVe?
  • In Table 1 of the appendix, why are the minimum scores for GPT-5.4 and GPT-5.4-mini so low? Also, why are the maximum scores for GPT-5.4 and GPT-5.4-mini so far below the maximum scores for the small open-weight models like Mistral, Phi4, and Llama3?
2 comments

Thank you for the thorough feedback, which helped us improve our paper, and for the list of additional references. Below we answer your specific comments and questions. See also our answers regarding common points, including adding the second task, in the general response.

Q1: Other embeddings.

The comparison with humans holds under all three embeddings; the ranking among the models changes (Tables 13 and 14). Word overlap, TTR and entropy use no embeddings and are therefore unaffected.

Q2, Q3, and W1: GPT scores compared with earlier studies and with open models.

We do not dispute the previous results; the difference comes from the instructions and the embedding. Like the human DAT, our prompts ban specialised vocabulary. The prompts of [4] and [1] do not, running essentially an easier task. These studies report the highest GPT scores; [5] keeps the specialized vocabulary restriction. With a minimal prompt that does not restrict the vocabulary, GPT-5.4 scores 91.91 and GPT-5.4-mini 86.48 (91.72 and 84.06 in [1]), yet GPT-5.4 keeps 18 times the human word overlap and uses "quasar" in 99 of 100 trials (Table 15). Under fastText and word2vec, all models score above the human mean and within two points of each other (Table 13). The GPT minima (67.0 and 66.9) are higher than those of the open models (56.6 to 63.6). The maxima are lower because the GPT models use common words: 0.3% and 0.0% of their tokens fall below Zipf 3, against 11.8% for Mistral and 21.6% for Llama3 (Tables 1 and 12).

W1: Aggregation.

Model values are means over conditions: 108 per open model, 72 for GPT-5.4 and 60 for GPT-5.4-mini (Section 2). The SDs in Table 1 are trial-level and include the spread between conditions (GPT-5.4: 72.3 to 83.1 across prompt and reasoning conditions; Table 24).

W1: Temperatures.

[1] samples at temperatures 1.0 to 2.0, which we include. Temperature 8, possible only for the local models, tests whether earlier attempts to gain diversity from temperature were limited by the range they tried and investigate how settings move one model's single-trial score and cross-trial diversity by taking an extreme value (Section 2). At temperature 1 alone the ordering of models is unchanged (Table 10).

W2. Only the DAT task in the original submission.

We agree that multiple tasks are more informative than one. The revision adds the Conditional DAT [2], compared against human responses to the same cues (Section 3.2). [1] found other tasks to be limited, including PACE, which mainly serves as a proxy for capability, not creativity. DRAT (Albinsson and Lief 2023), which, as far as we know, is not included in [1], is yet to be validated as a creativity test. LLMs’ novelty score on CDAT is near the human mean, yet two answers of one model to the same cue share 1.9 to 5.2 times as many words as two people's answers. Its appropriateness score covers the convergent component you emphasize. We also agree that the DAT is not a general creativity measure: while [1] finds it to be the best predictor of creative writing, no test predicts output diversity. Our cross-trial metrics complementing task scores aim to cover this gap.

Reviewer_C6FKOpenReview ↗

Reviewer Response

Thank you for your rebuttal and for adding the CDAT. After reading the other reviews, I maintain my score.

Reviewer_D4t4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether high scores on a creativity test also mean varied answers. It asks five language models to produce lists of ten words with different meanings. The authors repeat the task with different instructions and sampling settings. They measure test scores and repetition across answers, then compare the results with human responses. The models score near or above the human average but use a smaller range of words across repeated answers.

Critical strengths
  1. The paper asks an important question on the diversity of outputs of LLMs. A model that gives a strong answer once may still give many users similar answers. Measuring repetition can reveal something that an average score misses.

  2. The size of the repetition gap is striking, especially for the two GPT models. The broader finding that good individual answers can still be repetitive is less surprising.

Critical weaknesses
  1. The advance over earlier work is limited. NoveltyBench already showed that strong benchmark performance does not ensure varied answers across repeated generations (Zhang et al., 2025). Earlier work also measured diversity in vocabulary, sentence structure, and meaning (Guo et al., 2025), and showed that changing sampling settings can affect wording more than meaning (Tevet and Berant, 2021). This paper adds a detailed study on DAT, but the new insight beyond these findings remains limited.
  • Zhang et al. (2025). NoveltyBench: Evaluating Language Models for Humanlike Diversity.
  • Guo et al. (2025). Benchmarking Linguistic Diversity of Large Language Models.
  • Tevet and Berant (2021). Evaluating the Evaluation of Diversity in Natural Language Generation.
  1. The discussion sometimes moves from word repetition to claims about “genuine creativity” without explaining what supports that step. The experiments measure word choices on one task. They do not establish that models lack creativity in general or that reliable model behaviour prevents creative thinking.

  2. The paper compares repeated answers from one model with answers from many different people. This does not show whether a model produces less varied answers than one person completing the task repeatedly.

  3. The open models and GPT models use different sampling controls and temperature ranges. These differences make it difficult to separate the effect of the model from the effect of its settings.

  4. The authors remove a GPT-5.4-mini setting because it produces too many invalid responses, but they do not give an exact exclusion threshold or a full breakdown of failures. This makes it harder to assess the trade-off between diversity and valid answers.

  5. At least one reference omits author names (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena). Such errors suggest that the references need more careful checking, and that the manuscript was not prepared to professional academic standards.

Questions for authors
  1. How do you separate variation between different people from variation across repeated attempts, and how does this distinction affect your conclusions about human and model creativity?

  2. Does the finding that GPT models produce the least varied answers still hold when models are compared under matched sampling settings?

  3. When vocabulary size stays similar across prompts, are the models using the same words or different sets of words? What supports the claim that prompts mainly change how often existing words are used?

2 comments

Thank you for your helpful feedback on our paper. Points of general importance, and how we updated the paper, are addressed in the general response. Here we answer your specific comments and questions.

W1-W3. These are points of common interest. See our general response on how we addressed them.

W4. Unfortunately, OpenAI API restricts available decoding settings. We find it informative to report a broader range of decoding settings that open models support, including higher temperature values. We now report a comparison of all models at matched decoding settings (Table 10). See our response to Q2 below for more details.

W5: Exclusion threshold and breakdown of failures. We exclude a condition when more than one third of its responses are invalid. This applies to one condition, GPT-5.4-mini at temperature 2 and top-p 1.0. Failed attempts in the original runs were not saved; logged runs at the same settings give 35.7% invalid responses on the CDAT and 51 of 148 invalid attempts on the DAT with a minimal prompt. The criterion is stated in Section 2, and the excluded condition is reported separately (Table 16).

W6. References. We thank you for pointing out the error in one of the references. We corrected it and checked the other references.

Q1: Variation between people versus repeated attempts. The paper measures cross-trial diversity: how varied one system's answers are across independent, memoryless trials. This is what users who query one model repeatedly receive. The human sample is the reference because its responses are also independent. The human data hold one response per person, so we cannot say whether a model is less varied than one person answering repeatedly. We therefore claim that one system's independent trials are less varied than a population of independent responses, not that a model is less varied than one person repeating the task (Sections 1 and 4). See also the general response to reviews for our motivation for choosing this comparison setup.

Q2: Does the finding that GPT models produce the least varied answers still hold when models are compared under matched sampling settings? Yes, with a smaller gap. At temperature 1 and the widest sampling pool of each model class, both GPT models still have the highest word overlap and the lowest TTR. GPT-5.4-mini's overlap is 0.365 against Mistral's 0.329, compared with 0.508 against 0.239 over all conditions (Table 10).

Q3: Vocabulary across prompts. This differs by model. Open models largely reuse the same words across prompts (Llama3, Jaccard overlap within and between prompts: 0.40 and 0.33). GPT-5.4 uses a different narrow set of words for each prompt (0.64 and 0.32; Table 11). The claim you ask about therefore does not hold for all models. The revised paper states that a reasoning step changes word overlap but not TTR in both GPT models (Tables 21 and 22).

Reviewer_D4t4OpenReview ↗

Thank you.

Thank you for your response and the revisions. Some of my concerns have been addressed. My main concern is on the size and novelty of the contribution. It's already established that LLMs converge across repeated generations. This work confirms this by using a validated creativity test. The main finding is a known phenomenon measured on a new task. The paper does not offer a new explanation of why this convergence happens, or evidence that changes how the broader diversity literature should be read. The paper remains hard to follow after the revisions. I’d propose to the authors to do a substantial restructuring for their next version. In the current revised draft it remains hard to asses the contributions, have a clear understanding of the advancments versus the related work, and establish what results support each claim and answer each question differently than the prior work.

Reviewer_mQfMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

It askes several LLM to perform the DAT task. Comparing cross-trial diversity against samples of 100 human participants. All models score near or above the human DAT mean, yet show higher word overlap and much lower TTR.

Critical strengths
  • How diverse are LLMs and how well DAT measures it is an important question
  • The paper has proper statistical analysis with significant results
Critical weaknesses
  • I have issues with the paper's methodology and how it supports its claims. You compare running the same LLM (maybe with a different prompt, and seed) to do the DAT, to asking different humans. This doesn't answer your question about how LLM can generate diverse answers for a task. It does show that if everyone uses the same LLM we get similarity between uses. The paper mixes these two separate notions of diversity and needs to clarify its statement and to have proper experiments for the correct task it measures
  • How si entropy measured for humans (and closed source models) without GT probabilities. If it is estimated from that that can have a significant influence on results compared to likelihood from the logits themselves.
Questions for authors
  • How do you estimate entropy?
  • How is TTR effected by sample size?
  • How does this work differ from Wenger & Kenett (2025) and Haase et al. (2025)?
1 comment

Thank you for your thoughtful feedback, which helped us improve the paper. Below are answers to your comments and questions. They are also reflected in the general response.

W1:

Two notions of diversity. The aim of our work is to advance the assessment of creativity, which has an ever broadening relevance for diverse real-life applications. Both within-trial diversity, as measured in DAT scores, and diversity across trials, are aspects of creativity of immediate practical importance. We argue that both need to be assessed to derive practically useful measures of creativity, showing that these two types of diversity do not always go hand in hand even across setups involving the same language model. The revision offers a clarification (Section 1).

Human vs LLM comparison. We compare multiple runs of the same language model to a sample of responses from multiple humans. This setup is motivated by the practice of creative work, as users performing similar tasks are likely to independently use the same model, but much less likely to delegate the task to the same human writer. And even if that happens, a human can deliberately vary their responses to similar requests, an option not currently available to LLMs. This said, we added to Limitations a mention that the human sample provides a reference for independent responses, not repeated responses from one individual, and that the comparison is meant to inform the use of LLM vs human outputs in creative applications, not to assess LLMs as models of individual creativity.

W2 and Q1: How entropy is estimated.

No group's entropy uses GT probabilities or logits. Entropy is a plug-in estimate from observed word frequencies, then normalized by log2 of the number of unique words. The estimator is identical for humans and all models. For language models, it describes the words generated, not the next-token distribution of the model.

Q2: How is TTR affected by sample size?

All samples have the same size: 100 trials of ten words per model condition, and 100 participants per human bootstrap sample (Section 2). Smaller samples give a higher TTR for everyone and make the ratio between humans and models look smaller. To check this, we computed TTR from 10 to 100 trials (plots in Appendix A.4, Figure 18). Humans stay above every model at every size (10 trials: humans 0.92, Llama3 0.70, GPT-5.4 0.31; 100 trials: 0.64, 0.29, 0.08). The order of the models is stable from 30 trials on; word overlap does not depend on sample size.

Q3. How does this work differ from Wenger & Kenett (2025) and Haase et al. (2025)?

Wenger and Kenett compare LLMs with each other and with 102 people; their question is whether different models give similar answers. Haase et al. also run 100 DAT trials per model and prompt variant, but they report the spread of scores, not whether the same words recur. We measure convergence within one model, with four cross-trial metrics and a human reference of the same sample size, for five models (Section 1).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the reviews

We thank all reviewers for reading our paper and providing helpful feedback. The reviewers point out that we address an important question (all reviewers), and appreciate specific results we report (D4t4) and our statistical analysis (mQfM).

Clarifications

The reviewers’ comments and questions helped us further improve the paper. We clarified several points which the feedback highlighted:

  • Novelty relative to prior and concurrent work. The revised Introduction discusses our contributions more clearly and explicitly. We appreciate the reviewers’ constructive comments that helped us improve this aspect of the paper (D4t4’s W1, mQfM’s Q3; we also took into account all the literature suggested by C6FK).
  • Within- vs. cross-trial diversity. Reviewers highlight the need to clarify in the paper the relation between two types of diversity, within and across trials (mQfM), and the relevance of cross-trial diversity to creativity (D4t4, W2). We address both in the revised abstract and introduction, which clarify our argument that both types of diversity need to be measured for a practically useful assessment of creativity.
  • Why we compare repeated model answers with answers from different people. (mQfM, W1; D4t4, W3) We clarified this in the Introduction, adding the following:

This setup follows the practice of creative work: users performing similar tasks are likely to independently turn to the same model, but much less likely to delegate the task to the same human writer. Even when they do, a creative professional can deliberately vary their responses to similar requests, whereas a model answering in two independent sessions has no record of what it produced earlier.

We also added to Limitations that the human sample provides a reference for independent responses, not repeated responses from one individual, and that the comparison is meant to inform the use of LLM vs human outputs in creative applications, not to assess LLMs as models of individual creativity.

Additional results

The revised version reports additional empirical results motivated by the reviewers' input:

  • A second task: the Conditional DAT (C6FK). We added the CDAT, which has a set of human reference responses (Section 3.2, Figure 5, Tables 17 and 19). Schapiro et al. (2026) find the DAT to be the only test in their comparison with significant validity and specificity on all three creative-writing benchmarks. The CDAT adds a convergent component, and its appropriateness score correlates with creative writing and combinatorial creativity. Schapiro et al. found other tasks more limited; in particular, PACE largely serves as a proxy for model capability, which is distinct from creativity.
  • Two more embeddings (C6FK). We added DAT scoring under fastText and word2vec (Tables 13 and 14).
  • Matched decoding settings (D4t4). We report a comparison of all models at decoding settings shared across model classes (Table 10).
  • Exclusion criteria (D4t4). We clarified the exclusion criterion and report the excluded conditions separately (Section 2 and Table 16).
  • Discrepancy with previously reported GPT scores (C6FK). The previously reported high DAT scores for GPT models are replicated if we use a simplified prompt that differs from the instructions given to humans (see Section 3.1 and Table 15). Our main and original setup uses the same instructions as human DAT testing, which is essential for a meaningful LLM-to-human comparison.

Revised text is marked in blue. See also our responses to individual reviewers for more details and for answers to more specific questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.