acceptodds
Under review as a conference paper at ICLR 2027

Measuring Combinatorial Creativity in LLMs

Abstract

Combinatorial creativity, the process of combining familiar ideas in unfamiliar ways, is one of the foremost engines of scientific, technological, and artistic progress. While large language models (LLMs) can fluently produce novel text relative to their training data, their capacity to combine information in novel *and* useful ways, in order to invent new concepts, remains unclear. This work introduces **Kombine**, a combinatorial creativity benchmark tasking models to discover surprising, original, and useful associations, analogies, and blends between unrelated entities, and to use those analogies and blends to *invent novel concepts by re-combining the parts of existing ones*. In a large-scale study across 35 LLMs, we find that: **(1)** LLMs are 10-15 percentage points better at discovering associations and analogies than blends, and **(2)** exhibit an *abstraction bottleneck* on blending, in which even frontier LLMs struggle to find a shared abstraction between inputs and instantiate a coherent blended space. **(3)** LLMs also exhibit "inventive homogeneity," in which one in five concepts invented for a given item reuses properties invented by another model. Moreover, **(4)** models are more likely to reuse other invented properties on blends than analogies, and **(5)** models from the same provider are more likely to reuse properties than those from different providers. Our results reveal creative strengths in LLMs' capacity for association and analogy, point to an abstraction bottleneck on blending, and show how output homogeneity constrains the conceptual diversity of inventions. In summary, we provide a tool that can be used to assess progress towards more creative LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
creativity, large language models, analogy, blending, compositionality

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_A7nzOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary
  • introduces KOMBINE benchmark for comparing combinatorial capability of llms
  • takeaway: LLM show intutive homogenity, found new analogies, and reveal creative strengths in LLMs' capacity for association and analogy
  • scoring on useful, surpirsing and original
  • blending hardest task
Critical strengths
  • introduces a new benchmark
  • large scale study on LLMs
  • tool to measure creativeness
  • 3 domains, association, analogyy, blending
Critical weaknesses
  • how do you differentiate creativeness against factual incorrectness, truthfulness?

  • score is driven by validity: if not valid then 0, which doesnt aligns to interpret creativity

  • absrtaction bottleneck may come from judge end, which hasnt been explored

  • correlation of judge to human connection is thin

  • missing related works

  • Creativity tests on LLMs: Stevenson et al. (2022) on GPT-3 and the Alternate Uses Task; Koivisto & Grassini (2023), where AI beat average humans but not the best ones; Guzik et al. (2023) on the Torrance tests; Organisciak et al. (2023) on automated creativity scoring.

  • Judging creative quality: Chakrabarty et al. (CHI 2024), "Art or Artifice?" (TTCW); and Lu et al. (2024), "AI as Humanity's Salieri" (Creativity Index). The second measures novelty against human text on the web, which directly addresses this paper's weakest point.

  • AIs sounding alike: Doshi & Hauser (2024, Science Advances) on AI raising individual creativity but reducing diversity across people; Padmakumar & He (ICLR 2024) on writing with LLMs reducing diversity; Anderson et al. (2024) on homogenisation; Kirk et al. (2024) on RLHF reducing output diversity.

Questions for authors
2 comments
AuthorsOpenReview ↗

Author Rebuttal [1/2]

We sincerely thank the reviewer for the careful analysis of our work.

Weakness 1 (Differentiating creativity from factual incorrectness or truthfulness)

To answer this question and another related question poised by Reviewer En17, we conducted an analysis to distinguish between outputs obtained from genuinely creative reasoning versus outputs obtained by models regurgitating knowledge from their training data. To search for memorized inventions, we query the 4.6 trillion token OLMo2 pre-training mix from Infini-gram [5]. We define an (f,i,p)(f, i, p)-memorized invention as a concept whose name (e.g. “sun loaf”) occurs verbatim at least ff times in this pre-training corpora, whose name occurs within ii tokens of at least one of the two inputs to an item (e.g., “photosynthesis” and “bread”), and for which at least pp of the invention’s properties also occur within ‘i’ tokens of the concept’s name.

Provider (models)AnalogyBlendingAll
Meta (2)16.0%6.7%11.1%
Microsoft (1)6.7%5.6%6.2%
Qwen (2)3.3%5.0%4.2%
Moonshot (1)3.4%3.3%3.4%
Anthropic (10)1.7%2.4%2.0%
DeepSeek (3)1.1%1.1%1.1%
OpenAI (7)0.5%1.4%1.0%
Google (5)0.7%0.7%0.7%
xAI (2)000
Z.ai (2)000

Results. By these criteria, and for (f,i,p)=(10,100,2)(f, i, p) = (10, 100, 2), 43 of the 2,070 inventions (2.1%) are memorized. Unlike inventive multiples, models on average produce memorized concepts on analogy and blending at essentially equal rates (2.1% vs 2.0%). Furthermore, as shown above, the models which produce memorized inventions at the highest rate mostly come from the Meta, Microsoft, and Qwen families, with llama-3.3-70b at 17% (10 of 58), claude-sonnet-4.5 at 8% (5 of 60), qwen-2.5-72b at 7%, phi-4 at 6%, and llama-4-maverick at 5%. Conversely, 15 of 35 models never produce one. They include the current frontier: opus-5, fable-5, sonnet-5, sonnet-4.6, gpt-5.6-sol, gpt-5.2, gemini-3.1-pro, gemini-2.5-pro, deepseek-r1, and both Grok and both GLM models.

Overall, this analysis suggests that smaller, open-weight models are more likely to produce memorized concepts that rely on previously encountered relationships (and rely on factuality rather than creativity) than larger frontier models. We plan to incorporate this analysis into our revisions and want to thank you for raising this question.

Weakness 2 (Is Kombine robust to how scoring dimensions are aggregated?)

The utility gate operationalizes the standard two-criterion definition of a creative product, which must be novel and useful [1]. This follows a precedent in the creative psychometrics literature dating back to Maher (2010) which defines utility in terms of task-specific constraints [2]. Indeed, recent creativity tests for LLMs like the CDAT [3] and DRAT [4] have (appropriately) defined overall scoring as novelty conditioned on utility, following the precedent set by [2]. However, to assess the robustness of the benchmark to various other ways of aggregating task scores, we report the Spearman rank correlation between the default benchmark scoring and two alternative variants, as defined below.

Variant vs the paper rankingSpearman
No utility gate0.965
Geometric mean of task scores0.997

Results. Under two different aggregations (removing the utility gate on each item, or taking the geometric mean of each of the three task scores to compute the overall score), the spearman rank correlation with the default scoring remains very high (0.965 at worst). This indicates the ranking is robust to how scoring dimensions are aggregated, as well as whether utility is treated as a gate or as a separate dimension.

AuthorsOpenReview ↗

Author Rebuttal [2/2]

Weakness 3 (Is the abstraction bottleneck a robust finding?)

Multiple other reviewers suggested causally testing the abstraction bottleneck, a key finding which was previously only supported with correlational claims.

To do this, we took six models (claude-opus-5, gpt-5.2, deepseek-chat-v3-0324, llama-3.3-70b, qwen-2.5-72b, gpt-4o-mini) and assessed how well they could elaborate a coherent blended concept on items on which they previously failed by giving them valid generic spaces that other (often stronger) models discovered on those same items.

ModelBaselineGiven AbstractionAbstraction Bottleneck Gap
claude-opus-548.386.237.9pp
gpt-5.236.780.043.3pp
deepseek-chat-v3-032423.383.360.0pp
llama-3.3-70b30.063.333.3pp
qwen-2.5-72b26.760.033.3pp
gpt-4o-mini16.753.336.6pp
Average30.3±4.571.0±5.540.7±4.1 pp

Results: On blends a model previously failed, supplying a valid generic space from another model improves the rate at which models are able to elaborate a coherent blended space by 40.7 percentage points (pp) on average. For example, when asked to blend “Democracy” and “Banking,” gpt-4o-mini’s default generic space was “a financial system that allocates resources based on collective input,” which only applies to banking. However, when given a valid generic space (“a system converting personal endowments into collective claims through trusted agents”), gpt-4o-mini was then able to elaborate a coherent blended concept. In summary, the results from this experiment provide significant evidence that the abstraction bottleneck is a causal bottleneck to performance.

Weakness 4 (Validity of LLM Judges)

LLM judges agree with human raters as often as human raters agree with each other: We conducted a study where three of the authors independently and blindly rated 40 answers from 12 entity pairs to assess all judge-based scoring dimensions used on Kombine. This allowed us to assess how often two humans agree with each other (human-human agreement), versus how often the LLM judges' verdict (from majority vote) agrees with each human annotator, averaged over annotators (LLM–human agreement).

DimensionHuman–human agreementLLM–human agreement
Analogy: valid analogy69%72%
Analogy: mapping used44%67%
Analogy: coherence56%74%
Blending: generic space56%58%
Blending: coherent77%86%
Blending: scope44%43%

Results. On five of six subjectively scored dimensions, LLM-human agreement is better than human-human agreement. On analogy (validity, mapping used, coherence), as well as blending (generic space, blend coherence), the panel agrees with each rater at least as often as the other raters do. The one dimension where LLM-human agreement is slightly worse than human-human agreement is blending semantic integration quality, but LLM-human agreement is only 1 percentage point worse than human-human agreement here. Overall the LLM judge verdict on Kombine is as reliable as a single human annotator.

Weakness 5 (Missing related works)

Thank you for pointing these out! We will add them to the final draft.

References

  • [1] Runco, M. A., & Jaeger, G. J. (2012). The standard definition of creativity. Creativity research journal, 24(1), 92-96.
  • [2] Maher, Mary Lou. "Evaluating creativity in humans, computers, and collectively intelligent systems." Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design. 2010.
  • [3] Nakajima, Kumiko, Jan Zuiderveld, and Sandro Pezzelle. "Beyond divergent creativity: A human-based evaluation of creativity in large language models." Findings of the Association for Computational Linguistics: EACL 2026. 2026.
  • [4] Schapiro, S., Gladstone, A., Black, J., & Ji, H. (2026). Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers. arXiv preprint arXiv:2605.13450.
  • [5] Liu, J., Min, S., Zettlemoyer, L., Choi, Y., & Hajishirzi, H. (2024). Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377.
Reviewer_En17OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces KOMBINE, a benchmark for evaluating combinational creativity in LLMs, one of the three creativity types defined by Boden. The authors categorize combinational creativity into three forms: association, analogy, and conceptual blending. Unlike existing benchmarks that often focus on closed-ended recognition tasks, KOMBINE provides open-ended tasks requiring models to discover novel connections between unrelated entities and invent new concepts.

The authors evaluate 35 LLMs and identify several interesting findings, including an abstraction bottleneck in conceptual blending and a tendency for different models to generate similar inventions.

Critical strengths
  1. Well-motivated benchmark design. The paper provides a systematic framework for evaluating combinational creativity through three distinct mechanisms: association, analogy, and blending. Each task is formally defined with evaluation criteria, making the benchmark useful for assessing different aspects of creative reasoning. The open-ended setting is also well suited to evaluating creativity, as it requires discovering novel connections rather than recognizing predefined answers.

  2. Comprehensive evaluation and useful insights. The authors evaluate 35 LLMs, providing a broad comparison of their combinational creativity capabilities. The findings, particularly the abstraction bottleneck in conceptual blending and the homogeneity of invented concepts across models, offer useful insights into the current limitations of LLM creativity.

Critical weaknesses
  1. Limited human validation of creativity judgments.

Although the authors conduct human evaluation, the validation is relatively limited given that creativity is inherently subjective. In Appendix E, only 60 outputs are evaluated by a single human annotator. Furthermore, the agreement between human judgments and the LLM-panel majority is 66%, increasing to 75% when all three LLM judges agree.

While the authors provide additional inter-judge reliability and ranking robustness analyses, these results do not fully establish the validity of the LLM-based creativity evaluations. Expanding the human evaluation to include multiple independent annotators and more samples would strengthen confidence in the benchmark’s scoring methodology.

  1. Limited validation of the abstraction bottleneck conclusion.

One of the paper’s central findings is that abstraction is the main bottleneck in conceptual blending. The authors support this conclusion by showing that models frequently fail to construct valid generic spaces but usually generate coherent blended concepts when a valid generic space is found.

However, the validity of these generic spaces and the coherence of the resulting concepts are assessed using LLM judges. Given the limited human agreement, it is unclear how robust this conclusion is to human evaluation.

Additionally, the conditional analysis does not establish whether abstraction is the primary causal bottleneck. An experiment supplying validated generic spaces and measuring improvements in downstream blending performance would provide stronger evidence for this claim.

Questions for authors
  1. Data contamination and memorization: How do the authors distinguish genuinely novel combinations from associations or concepts that models may have memorized during pretraining? Although the benchmark uses unrelated entity pairs, could models rely on previously encountered relationships or inventions? Have the authors considered additional controls to isolate the contribution of creative reasoning from memorized knowledge?

  2. Abstraction bottleneck: Have the authors considered evaluating blending performance when models are explicitly provided with human-validated generic spaces? This could help determine whether abstraction is indeed the primary bottleneck or whether other stages of conceptual blending contribute substantially to failures.

2 comments
AuthorsOpenReview ↗

Author Rebuttal [1/2]

We are very grateful for these excellent suggestions and for this careful review of our work.

Weakness 2 / Question 2 (Is the abstraction bottleneck causal?)

Multiple reviewers (both yourself and cGKy) suggested causally testing the abstraction bottleneck, a key finding which was previously only supported with correlational claims.

To do this, we took six models (claude-opus-5, gpt-5.2, deepseek-chat-v3-0324, llama-3.3-70b, qwen-2.5-72b, gpt-4o-mini) and assessed how well they could elaborate a coherent blended concept on items on which they previously failed by giving them valid generic spaces that other (often stronger) models discovered on those same items.

ModelBaselineGiven AbstractionAbstraction Bottleneck Gap
claude-opus-548.386.237.9pp
gpt-5.236.780.043.3pp
deepseek-chat-v3-032423.383.360.0pp
llama-3.3-70b30.063.333.3pp
qwen-2.5-72b26.760.033.3pp
gpt-4o-mini16.753.336.6pp
Average30.3±4.571.0±5.540.7±4.1 pp

Results: On blends a model previously failed, supplying a valid generic space from another model improves the rate at which models are able to elaborate a coherent blended space by 40.7 percentage points (pp) on average. For example, when asked to blend “Democracy” and “Banking,” gpt-4o-mini’s default generic space was “a financial system that allocates resources based on collective input,” which only applies to banking. However, when given a valid generic space (“a system converting personal endowments into collective claims through trusted agents”), gpt-4o-mini was then able to elaborate a coherent blended concept. In summary, the results from this experiment provide significant evidence that the abstraction bottleneck is a causal bottleneck to performance.

Weakness 1 (Do the LLM judges align with human ratings of creativity?)

Two reviewers (yourself and A7nz) suggested expanding the human evaluation to multiple independent annotators. To do this, we conducted an additional study where three of the authors independently and blindly rated 40 answers from 12 entity pairs to assess all judge-based scoring dimensions used on Kombine. This allowed us to assess how often two humans agree with each other (human-human agreement), versus how often the LLM judges' verdict (from majority vote) agrees with each human annotator, averaged over annotators (LLM–human agreement).

DimensionHuman–human agreementLLM–human agreement
Analogy: valid analogy69%72%
Analogy: mapping used44%67%
Analogy: coherence56%74%
Blending: generic space56%58%
Blending: coherent77%86%
Blending: integration quality44%43%

Results. On five of six subjectively scored dimensions, LLM-human agreement is better than human-human agreement. On analogy (validity, mapping used, coherence), as well as blending (generic space, blend coherence), the panel agrees with each rater at least as often as the other raters do. The one dimension where LLM-human agreement is slightly worse than human-human agreement is blending semantic integration quality, but LLM-human agreement is only 1 percentage point worse than human-human agreement here. Overall the LLM judge verdict on Kombine is as reliable as a single human annotator.

AuthorsOpenReview ↗

Author Rebuttal [2/2]

Question 1 (How do you distinguish creative reasoning from memorized knowledge?)

We are grateful for this timely and important question. To search for memorized inventions, we query the 4.6 trillion token OLMo2 pre-training mix from Infini-gram [1]. We define an (f,i,p)(f, i, p)-memorized invention as a concept whose name (e.g. “sun loaf”) occurs verbatim at least ff times in this pre-training corpora, whose name occurs within ii tokens of at least one of the two inputs to an item (e.g., “photosynthesis” and “bread”), and for which at least pp of the invention’s properties also occur within ‘i’ tokens of the concept’s name.

Provider (models)AnalogyBlendingAll
Meta (2)16.0%6.7%11.1%
Microsoft (1)6.7%5.6%6.2%
Qwen (2)3.3%5.0%4.2%
Moonshot (1)3.4%3.3%3.4%
Anthropic (10)1.7%2.4%2.0%
DeepSeek (3)1.1%1.1%1.1%
OpenAI (7)0.5%1.4%1.0%
Google (5)0.7%0.7%0.7%
xAI (2)000
Z.ai (2)000

Results. By these criteria, and for (f,i,p)=(10,100,2)(f, i, p) = (10, 100, 2), 43 of the 2,070 inventions (2.1%) are memorized. Unlike inventive multiples, models on average produce memorized concepts on analogy and blending at essentially equal rates (2.1% vs 2.0%). Furthermore, as shown above, the models which produce memorized inventions at the highest rate mostly come from the Meta, Microsoft, and Qwen families, with llama-3.3-70b at 17% (10 of 58), claude-sonnet-4.5 at 8% (5 of 60), qwen-2.5-72b at 7%, phi-4 at 6%, and llama-4-maverick at 5%. Conversely, 15 of 35 models never produce one. They include the current frontier: opus-5, fable-5, sonnet-5, sonnet-4.6, gpt-5.6-sol, gpt-5.2, gemini-3.1-pro, gemini-2.5-pro, deepseek-r1, and both Grok and both GLM models.

Overall, this analysis suggests that smaller, open-weight models are more likely to produce memorized concepts that rely on previously encountered relationships (as you suggest), than larger frontier models. We plan to incorporate this analysis into our revisions and want to thank you once again for raising this question.

[1] Liu, J., Min, S., Zettlemoyer, L., Choi, Y., & Hajishirzi, H. (2024). Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377.

Reviewer_cGKyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KOMBINE probes combinatorial creativity in LLMs via three cognitive-science tasks: association (find a chain of true triples linking two unrelated entities), analogy (find two parallel paths giving a structure mapping, then invent a concept by projecting a source entity), and blending (find a shared generic space, then build a blended concept with emergent properties). Entities come from Wikidata: 283 across 21 domains, each with >= 78 sitelinks. Outputs are scored on utility, surprise and originality, plus emergent creativity of the invented concept for analogy and blending. Utility is judged by LLMs (Claude Haiku 4.5 for factuality, a three-model panel otherwise); surprise is embedding distance; originality is mean cosine distance to the nearest neighbours among all outputs the 35 models produced for the same item. Reported results: valid outputs are most common for association and analogy, rarer for blends (59.1% / 54.0% / 43.8%); an "abstraction bottleneck" appears, since a coherent blend follows a valid generic space 97.1% of the time; ~20% of invented concepts are "inventive multiples" (reusing >= 2 properties of another model's concept at cosine 0.674); blends produce multiples far more than analogies (14.3x at λ=2); same-provider pairs produce multiples 2.3x more than cross-provider pairs.

Critical strengths
  1. Well-motivated, open-ended benchmark with an explicit cognitive-science grounding, and clearly written. Prior work mostly tests recognition of known analogies or associations; KOMBINE makes models discover connections between unrelated entities and then use them, with the structure made explicit (paths, mappings, generic spaces, projected vs emergent properties). Covering 35 models including recent frontier systems gives a useful snapshot, and the robustness checks are sensible (rankings across embedding models, ranking on judge-unanimous items only, ρ = 0.945, bootstrap CIs).
  2. The homogeneity analysis is the most interesting angle. Looking at overlap between models, not just per-model scores, and asking whether it is stronger within a provider, is a question the community cares about. The per-provider breakdown is informative (DeepSeek and Anthropic +12.2 and +11.1 points above baseline, OpenAI slightly below).
Critical weaknesses
  1. Construct validity of the scores is not established. Scores depend on the model pool and unvalidated judges.
  • Originality is defined against the evaluated pool, so one model's score shifts when models are added or removed (a near-duplicate lowers both models' originality). This is benchmark-relative, not novelty against human knowledge, as the limitations section partly admits. It is also partly circular with the homogeneity result: the metric rewards being unlike the others, and the paper then reports that models are alike. No leave-one-model-out or fixed-reference-pool check appears.
  • Surprise is embedding distance (all-MiniLM-L6-v2). Semantic distance between entities is not surprise. The overall score is also an equal-weight composite with a hard zero on utility failure, and neither choice is justified or tested.
  • Judges: path factuality is judged by Claude Haiku 4.5, plausibly weaker than many models it judges, with no retrieval. The panel includes evaluated models (self-preference?). Panel reliability is only "fair to good" (ICC(2,3) 0.48–0.67). The human check is one author rating 60 items: 66% agreement with the panel majority, 75% on the unanimous subset only, and the main text quotes the 75%.
  • No human baseline or human creativity ratings, so we cannot tell whether higher scores mean "more creative" to anyone.
  • The headline findings are over-interpreted given how they are measured.

Key findings rest on non-comparable measures and arbitrary thresholds.

  • "Blending is 10–15 points harder" and the "abstraction bottleneck": the three validity rates use different criteria and judges per task (path factuality; mapping validity; generic-space fit), so they are not on a common scale. A lower blend rate may reflect a stricter rubric as much as a harder ability. The 97.1% figure is conditional on that gate and judged by a different rubric. The proposed explanation (missing experiential knowledge) is speculation with no test, e.g. providing the generic space or a hint.
  • Inventive multiples: the 0.674 threshold is the 99th percentile of all property-pair similarities, so ~1% of pairs are multiples by construction, and results depend on this choice (no sensitivity analysis seen). Properties are short relation–object fragments embedded with MiniLM. The 14.3× blend-vs-analogy comparison may be confounded by properties per concept and by task structure (blend properties are constrained by a shared generic space, so similar wording is likely). No baseline for "homogeneous": how often do two samples from the same model at T = 0.9 form multiples?
  • Same-provider effect (9.3% vs 4.1%): a provider's models include successive versions that likely share training data, so this could be version similarity rather than "style". Separating near-versions from genuinely different models would help.
  • Size: from the numbers (2,070 inventions across 35 models, 283 entities) the benchmark looks like roughly 30 entity pairs per task. Please confirm. That is small for per-model conclusions, and the number of samples per item at T = 0.9 is not stated.
Questions for authors
  1. How many entity pairs per task, and how many samples per model per item? How much does a model's score vary across resamples at T = 0.9?
  2. How stable is the leaderboard, originality especially, if you remove a model or a whole provider, or use a fixed reference pool?
  3. Do surprise and originality correlate with human creativity judgements? Even a modest study with several raters on a few hundred items would help far more than the current 60 items from one rater.
  4. For multiples: what is the rate between two samples of the same model, how sensitive are the results to the threshold (95th / 99th / 99.9th percentile), and are they robust when normalised for properties per concept?
  5. How was the Haiku 4.5 factuality judge validated? What is its error rate on paths from stronger models, and is there self-preference in the panel?
  6. Since reasoning level is left at default and differs across models, how much of the ranking reflects thinking budget rather than ability?
4 comments
AuthorsOpenReview ↗

Author Rebuttal [1/3]

We sincerely thank the reviewer for this detailed review of our paper and for the constructive comments.

Weakness 1 (bullet 1) / Question 2 (Is Kombine robust to originality scoring?)

Since originality is defined relative to the pool of outputs given by other models, we assessed the robustness of originality scoring by recomputing this dimension under two alternative pools:

  • Leave one model out (n=35 configurations)
  • Leave one model provider (e.g. Anthropic) out (n=10 configurations)
Originality scoring variantnOverall Spearman (min)
Leave one model out351.000
Leave one provider out100.9996

Results. Kombine is highly robust to how originality is scored. Leaving out any single model leaves the ranking of the remaining 34 completely unchanged (the Spearman rank correlation to the default ranking is 1.00 in all 35 cases). Removing a whole provider, including all 10 Anthropic or all 7 OpenAI models, leaves the ranking essentially unchanged (the Spearman rank correlation drops to at most 0.9996 over all 10 cases).

Weakness 1 (bullet 2) (Is Kombine robust to how scoring dimensions are aggregated?)

The utility gate operationalizes the standard two-criterion definition of a creative product, which must be novel and useful. This follows a precedent in the creative psychometrics literature dating back to Maher (2010) which defines utility in terms of task-specific constraints [2]. Indeed, recent creativity tests for LLMs like the CDAT [3] and DRAT [4] have (appropriately) defined overall scoring as novelty conditioned on utility, following the precedent set by [2]. However, to assess the robustness of the benchmark to various other ways of aggregating task scores, we report the Spearman rank correlation between the default benchmark scoring and two alternative variants, as defined below.

Variant vs the paper rankingSpearman
No utility gate0.965
Geometric mean of task scores0.997

Results. Under two different aggregations (removing the utility gate on each item, or taking the geometric mean of each of the three task scores to compute the overall score), the spearman rank correlation with the default scoring remains very high (0.965 at worst). This indicates the ranking is robust to how scoring dimensions are aggregated, as well as whether utility is treated as a gate or as a separate dimension.

Regarding our measurement of surprise: semantic distance between entities is a common and well-established method to assess surprise in the computational creativity literature [5, 6], as it is asymptotically equivalent to Bayesian surprise [5, 7]. For example, the Forward Flow measure [8], the Divergent Semantic Integration (DSI) metric [6], and the creativity evaluation framework in [5] all measure surprise or creativity using semantic distance between entities, and [6, 8] have been validated against human judgments.

References

  • [1] Liu, J., Min, S., Zettlemoyer, L., Choi, Y., & Hajishirzi, H. (2024). Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377.
  • [2] Maher, Mary Lou. "Evaluating creativity in humans, computers, and collectively intelligent systems." Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design. 2010.
  • [3] Nakajima, Kumiko, Jan Zuiderveld, and Sandro Pezzelle. "Beyond divergent creativity: A human-based evaluation of creativity in large language models." Findings of the Association for Computational Linguistics: EACL 2026. 2026.
  • [4] Schapiro, S., Gladstone, A., Black, J., & Ji, H. (2026). Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers. arXiv preprint arXiv:2605.13450.
  • [5] Varshney, L. R., Rajani, N. F., & Socher, R. (2020). Explaining creative artifacts. arXiv preprint arXiv:2010.07126.
  • [6] Johnson, D. R., Kaufman, J. C., Baker, B. S., Patterson, J. D., Barbot, B., Green, A. E., ... & Beaty, R. E. (2023). Divergent semantic integration (DSI): Extracting creativity from narratives with distributional semantic modeling. Behavior research methods, 55(7), 3726-3759.
  • [7] Varshney, L. R. (2019). Mathematical limit theorems for computational creativity. IBM Journal of Research and Development, 63(1), 2-1.
  • [8] Gray, Kurt, Stephen Anderson, Eric Evan Chen, John Michael Kelly, Michael S. Christian, John Patrick, Laura Huang, Yoed N. Kenett, and Kevin Lewis. "“Forward flow”: A new measure to quantify free thought and predict creativity." American Psychologist 74, no. 5 (2019): 539.
AuthorsOpenReview ↗

Author Rebuttal [2/3]

Weakness 1 (bullets 3 and 4) / Question 3 (Validity of LLM judges)

Kombine is robust to judge “family-preference bias”: To start, we want to clarify that none of the three LLM judges (claude-haiku-4.5, gpt-5.4, o3) is among the 35 evaluated models, so self-preference bias does not apply here. However, judges do share providers with evaluated models (10 Anthropic and 7 OpenAI models among the 35), so we assessed whether “family-preference” bias exists by recomputing the leaderboard for each judge individually.

PanelSpearman ρ
Haiku only0.978
o3 only0.991
gpt-5.4 only0.982

Results. Restricting to any individual judge leaves the ranking nearly essentially intact, with the revised leaderboard correlating at worst ρ=0.978\rho = 0.978 with the default leaderboard, suggesting that family-preference bias, if at all, is minimal and does not affect the results.

LLM judges agree with human raters as often as human raters agree with each other: Second, we conducted an additional study where three of the authors independently and blindly rated 40 answers from 12 entity pairs to assess all judge-based scoring dimensions used on Kombine. This allowed us to assess how often two humans agree with each other (human-human agreement), versus how often the LLM judges' verdict (from majority vote) agrees with each human annotator, averaged over annotators (LLM–human agreement).

DimensionHuman–human agreementLLM–human agreement
Analogy: valid analogy69%72%
Analogy: mapping used44%67%
Analogy: coherence56%74%
Blending: generic space56%58%
Blending: coherent77%86%
Blending: scope44%43%

Results. On five of six subjectively scored dimensions, LLM-human agreement is better than human-human agreement. On analogy (validity, mapping used, coherence), as well as blending (generic space, blend coherence), the panel agrees with each rater at least as often as the other raters do. The one dimension where LLM-human agreement is slightly worse than human-human agreement is blending semantic integration quality, but LLM-human agreement is only 1 percentage point worse than human-human agreement here. Overall the LLM judge verdict on Kombine is as reliable as a single human annotator.

Weakness 2 (bullet point 4) / Question 1 (Kombine has high test-retest reliability)

Kombine scores are obtained by sampling models at T = 0.9 for 30 input pairs per task. Below, we assess test-retest reliability under 30 new input pairs that share no entities with the original 30. We find that overall scores on Kombine remain largely unchanged (Spearman ρ\rho = 0.88), with the analogy and blending task remaining highly reliable (Spearman ρ\rho = 0.88 and 0.94, respectively). However, the association task has the lowest test-retest reliability (Spearman ρ\rho = 0.71), but this is still considered good. In summary, the overall scores remain largely unchanged using 30 distinct items.

ScorePearson rSpearman ρ\rho**Mean $\Delta$ (points)**
Association0.650.717.1
Analogy0.870.887.8
Blending0.950.945.8
Overall0.920.882.7

Weakness 2 (bullet 2) / Question 4 (Inventive multiples comparative analyses are robust to θ\theta)

To assess the robustness of the inventive multiple analysis, we swept the inventive multiple threshold from eight values between 0.55 and 0.75. The table below reports the multiple rate at τ=2\tau = 2. As shown, the blend-to-analogy multiple ratio remains roughly the same across all values (12-14x), and the same-to-different provider multiple ratio stays between 1.6-2.3x. Finally, normalized the blend-to-analogy multiple ratio by the number of properties in each, the normalized blend-to-analogy ratio remains roughly 3.2-3.6x—in other words, even though blends tend to have more properties, they still homogenize at a 3.2-3.6x greater rate than analogies when controlling for the number of properties.

θOverallBlendsAnalogiesRatioPer propertyProvider
0.5545.3%83.7%6.9%12.1×3.3×1.6×
0.6034.9%65.1%4.9%13.2×3.2×1.9×
0.6230.8%57.3%4.3%13.2×3.3×2.1×
0.6524.8%45.9%3.8%12.2×3.3×2.3×
0.674 (paper)19.9%37.3%2.6%14.3×3.4×2.3×
0.7014.8%27.5%2.2%12.4×3.5×2.1×
0.7211.7%21.9%1.5%14.2×3.5×2.0×
0.758.2%15.2%1.2%13.1×3.6×1.8×
AuthorsOpenReview ↗

Author Rebuttal [3/3]

Question 6 (Performance on Kombine is agnostic to reasoning effort)

To assess how performance on Kombine is affected by reasoning effort, we increased reasoning effort on two frontier models (gpt-5.6-sol and gpt-6-astra). Although reasoning tokens increase nearly 7-14x from low-high, there is no significant difference in overall performance on each model, as overall scores on medium and high fall within the 95% confidence interval of scores on low reasoning effort. In fact, for both sol and astra, high reasoning effort leads to lower performance than low reasoning effort, despite using 7-14x as many tokens.

Model · effortReasoning tokensAssociationAnalogyBlendingOverall [95% CI]
sol · low1,78649.178.663.863.8 [59.1, 67.9]
sol · medium4,98146.771.867.261.9 [56.7, 66.5]
sol · high13,076 (7.3×)44.573.971.963.4 [58.3, 67.8]
astra · low1,01144.970.064.959.9 [54.3, 65.0]
astra · medium6,52644.466.453.854.8 [48.6, 60.3]
astra · high14,779 (14.6×)46.670.361.659.5 [53.6, 64.5]

Thanks for your detailed response, which addressed my concerns. I will increase my score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Author Rebuttal Experiments [1/4]

We thank all three reviewers for providing detailed reviews and helping us improve our work. Below, we summarize the extensive experiments we have conducted during the rebuttal period.

1. Is the abstraction bottleneck causal? (En17 Weakness 2 / Q2, A7nz Weakness 3, cGKy Weakness 2)

Multiple reviewers (both cGKy and En17) suggested causally testing the abstraction bottleneck, a key finding which was previously only supported with correlational claims.

To do this, we took six models (claude-opus-5, gpt-5.2, deepseek-chat-v3-0324, llama-3.3-70b, qwen-2.5-72b, gpt-4o-mini) and assessed how well they could elaborate a coherent blended concept on items on which they previously failed by giving them valid generic spaces that other (often stronger) models discovered on those same items.

ModelBaselineGiven AbstractionAbstraction Bottleneck Gap
claude-opus-548.386.237.9pp
gpt-5.236.780.043.3pp
deepseek-chat-v3-032423.383.360.0pp
llama-3.3-70b30.063.333.3pp
qwen-2.5-72b26.760.033.3pp
gpt-4o-mini16.753.336.6pp
Average30.3±4.571.0±5.540.7±4.1 pp

Results: On blends a model previously failed, supplying a valid generic space from another model improves the rate at which models are able to elaborate a coherent blended space by 40.7 percentage points (pp) on average. For example, when asked to blend “Democracy” and “Banking,” gpt-4o-mini’s default generic space was “a financial system that allocates resources based on collective input,” which only applies to banking. However, when given a valid generic space (“a system converting personal endowments into collective claims through trusted agents”), gpt-4o-mini was then able to elaborate a coherent blended concept. In summary, the results from this experiment provide significant evidence that the abstraction bottleneck is a causal bottleneck to performance.

2. Validity of LLM judges (cGKy Weakness 1 bullets 3–4 / Q3 / Q5, En17 Weakness 1, A7nz Weakness 4)

Kombine is robust to judge “family-preference bias”: To start, we want to clarify that none of the three LLM judges (claude-haiku-4.5, gpt-5.4, o3) is among the 35 evaluated models, so self-preference bias does not apply here. However, judges do share providers with evaluated models (10 Anthropic and 7 OpenAI models among the 35), so we assessed whether “family-preference” bias exists by recomputing the leaderboard for each judge individually.

PanelSpearman ρ\rho
Haiku only0.978
o3 only0.991
gpt-5.4 only0.982

Results. Restricting to any individual judge leaves the ranking nearly essentially intact, with the revised leaderboard correlating at worst ρ=0.978\rho = 0.978 with the default leaderboard, suggesting that family-preference bias, if at all, is minimal and does not affect the results.

AuthorsOpenReview ↗

Summary of Author Rebuttal Experiments [2/4]

LLM judges agree with human raters as often as human raters agree with each other: Second, we conducted an additional study where three of the authors independently and blindly rated 40 answers from 12 entity pairs to assess all judge-based scoring dimensions used on Kombine. This allowed us to assess how often two humans agree with each other (human-human agreement), versus how often the LLM judges' verdict (from majority vote) agrees with each human annotator, averaged over annotators (LLM–human agreement).

DimensionHuman–human agreementLLM–human agreement
Analogy: valid analogy69%72%
Analogy: mapping used44%67%
Analogy: coherence56%74%
Blending: generic space56%58%
Blending: coherent77%86%
Blending: scope44%43%

Results. On five of six subjectively scored dimensions, LLM-human agreement is better than human-human agreement. On analogy (validity, mapping used, coherence), as well as blending (generic space, blend coherence), the panel agrees with each rater at least as often as the other raters do. The one dimension where LLM-human agreement is slightly worse than human-human agreement is blending semantic integration quality, but LLM-human agreement is only 1 percentage point worse than human-human agreement here. Overall the LLM judge verdict on Kombine is as reliable as a single human annotator.

3. How do you distinguish creative reasoning from memorized knowledge? (En17 Q1, A7nz Weakness 1)

To distinguish between outputs obtained from genuinely creative reasoning versus outputs obtained by models regurgitating knowledge from their training data, we conducted an analysis to search for memorized inventions. To do so, we query the 4.6 trillion token OLMo2 pre-training mix from Infini-gram [1]. We define an (f,i,p)(f, i, p)-memorized invention as a concept whose name (e.g. “sun loaf”) occurs verbatim at least ff times in this pre-training corpora, whose name occurs within ii tokens of at least one of the two inputs to an item (e.g., “photosynthesis” and “bread”), and for which at least pp of the invention’s properties also occur within ‘i’ tokens of the concept’s name.

Provider (models)AnalogyBlendingAll
Meta (2)16.0%6.7%11.1%
Microsoft (1)6.7%5.6%6.2%
Qwen (2)3.3%5.0%4.2%
Moonshot (1)3.4%3.3%3.4%
Anthropic (10)1.7%2.4%2.0%
DeepSeek (3)1.1%1.1%1.1%
OpenAI (7)0.5%1.4%1.0%
Google (5)0.7%0.7%0.7%
xAI (2)000
Z.ai (2)000

Results. By these criteria, and for (f,i,p)=(10,100,2)(f, i, p) = (10, 100, 2), 43 of the 2,070 inventions (2.1%) are memorized. Unlike inventive multiples, models on average produce memorized concepts on analogy and blending at essentially equal rates (2.1% vs 2.0%). Furthermore, as shown above, the models which produce memorized inventions at the highest rate mostly come from the Meta, Microsoft, and Qwen families, with llama-3.3-70b at 17% (10 of 58), claude-sonnet-4.5 at 8% (5 of 60), qwen-2.5-72b at 7%, phi-4 at 6%, and llama-4-maverick at 5%. Conversely, 15 of 35 models never produce one. They include the current frontier: opus-5, fable-5, sonnet-5, sonnet-4.6, gpt-5.6-sol, gpt-5.2, gemini-3.1-pro, gemini-2.5-pro, deepseek-r1, and both Grok and both GLM models.

Overall, this analysis suggests that smaller, open-weight models are more likely to produce memorized concepts that rely on previously encountered relationships (and rely on factuality rather than creativity) than larger frontier models. We plan to incorporate this analysis into our revisions.

AuthorsOpenReview ↗

Summary of Author Rebuttal Experiments [3/4]

4. Is Kombine robust to design choices?

4a. Is Kombine robust to originality scoring? (cGKy Weakness 1 bullet 1 / Q2)

Since originality is defined relative to the pool of outputs given by other models, we assessed the robustness of originality scoring by recomputing this dimension under two alternative pools:

  • Leave one model out (n=35 configurations)
  • Leave one model provider (e.g. Anthropic) out (n=10 configurations)
Originality scoring variantnOverall Spearman (min)
Leave one model out351.000
Leave one provider out100.9996

Results. Kombine is highly robust to how originality is scored. Leaving out any single model leaves the ranking of the remaining 34 completely unchanged (the Spearman rank correlation to the default ranking is 1.00 in all 35 cases). Removing a whole provider, including all 10 Anthropic or all 7 OpenAI models, leaves the ranking essentially unchanged (the Spearman rank correlation drops to at most 0.9996 over all 10 cases).

4b. Is Kombine robust to how scoring dimensions are aggregated? (cGKy Weakness 1 bullet 2, A7nz Weakness 2)

The utility gate operationalizes the standard two-criterion definition of a creative product, which must be novel and useful [2]. This follows a precedent in the creative psychometrics literature dating back to Maher (2010) which defines utility in terms of task-specific constraints [3]. Indeed, recent creativity tests for LLMs like the CDAT [4] and DRAT [5] have (appropriately) defined overall scoring as novelty conditioned on utility, following the precedent set by [3]. However, to assess the robustness of the benchmark to various other ways of aggregating task scores, we report the Spearman rank correlation between the default benchmark scoring and two alternative variants, as defined below.

Variant vs the paper rankingSpearman
No utility gate0.965
Geometric mean of task scores0.997

Results. Under two different aggregations (removing the utility gate on each item, or taking the geometric mean of each of the three task scores to compute the overall score), the spearman rank correlation with the default scoring remains very high (0.965 at worst). This indicates the ranking is robust to how scoring dimensions are aggregated, as well as whether utility is treated as a gate or as a separate dimension.

Regarding our measurement of surprise: semantic distance between entities is a common and well-established method to assess surprise in the computational creativity literature [6, 7], as it is asymptotically equivalent to Bayesian surprise [6, 8]. For example, the Forward Flow measure [9], the Divergent Semantic Integration (DSI) metric [7], and the creativity evaluation framework in [6] all measure surprise or creativity using semantic distance between entities, and [7, 9] have been validated against human judgments.

4c. Kombine has high test-retest reliability (cGKy Weakness 2 bullet 4 / Q1)

Kombine scores are obtained by sampling models at T = 0.9 for 30 input pairs per task. Below, we assess test-retest reliability under 30 new input pairs that share no entities with the original 30. We find that overall scores on Kombine remain largely unchanged (Spearman ρ\rho = 0.88), with the analogy and blending task remaining highly reliable (Spearman ρ\rho = 0.88 and 0.94, respectively). However, the association task has the lowest test-retest reliability (Spearman ρ\rho = 0.71), but this is still considered good. In summary, the overall scores remain largely unchanged using 30 distinct items.

ScorePearson rSpearman ρ\rhoMean |Δ| (points)
Association0.650.717.1
Analogy0.870.887.8
Blending0.950.945.8
Overall0.920.882.7
AuthorsOpenReview ↗

Summary of Author Rebuttal Experiments [4/4]

4d. Performance on Kombine is agnostic to reasoning effort (cGKy Q6)

To assess how performance on Kombine is affected by reasoning effort, we increased reasoning effort on two frontier models (gpt-5.6-sol and gpt-6-astra). Although reasoning tokens increase nearly 7-14x from low-high, there is no significant difference in overall performance on each model, as overall scores on medium and high fall within the 95% confidence interval of scores on low reasoning effort. In fact, for both sol and astra, high reasoning effort leads to lower performance than low reasoning effort, despite using 7-14x as many tokens.

Model · effortReasoning tokensAssociationAnalogyBlendingOverall [95% CI]
sol · low1,78649.178.663.863.8 [59.1, 67.9]
sol · medium4,98146.771.867.261.9 [56.7, 66.5]
sol · high13,076 (7.3×)44.573.971.963.4 [58.3, 67.8]
astra · low1,01144.970.064.959.9 [54.3, 65.0]
astra · medium6,52644.466.453.854.8 [48.6, 60.3]
astra · high14,779 (14.6×)46.670.361.659.5 [53.6, 64.5]

4e. Inventive multiples comparative analyses are robust to θ\theta (cGKy Weakness 2 bullet 2 / Q4)

To assess the robustness of the inventive multiple analysis, we swept the inventive multiple threshold from eight values between 0.55 and 0.75. The table below reports the multiple rate at τ=2\tau = 2. As shown, the blend-to-analogy multiple ratio remains roughly the same across all values (12-14x), and the same-to-different provider multiple ratio stays between 1.6-2.3x. Finally, normalized the blend-to-analogy multiple ratio by the number of properties in each, the normalized blend-to-analogy ratio remains roughly 3.2-3.6x—in other words, even though blends tend to have more properties, they still homogenize at a 3.2-3.6x greater rate than analogies when controlling for the number of properties.

θ\thetaOverallBlendsAnalogiesRatioPer propertyProvider
0.5545.3%83.7%6.9%12.1×3.3×1.6×
0.6034.9%65.1%4.9%13.2×3.2×1.9×
0.6230.8%57.3%4.3%13.2×3.3×2.1×
0.6524.8%45.9%3.8%12.2×3.3×2.3×
0.674 (paper)19.9%37.3%2.6%14.3×3.4×2.3×
0.7014.8%27.5%2.2%12.4×3.5×2.1×
0.7211.7%21.9%1.5%14.2×3.5×2.0×
0.758.2%15.2%1.2%13.1×3.6×1.8×

5. Missing related works (A7nz Weakness 5)

Thank you for pointing these out! We will add them to the final draft.

References

  • [1] Liu, J., Min, S., Zettlemoyer, L., Choi, Y., & Hajishirzi, H. (2024). Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377.
  • [2] Runco, M. A., & Jaeger, G. J. (2012). The standard definition of creativity. Creativity research journal, 24(1), 92-96.
  • [3] Maher, Mary Lou. "Evaluating creativity in humans, computers, and collectively intelligent systems." Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design. 2010.
  • [4] Nakajima, Kumiko, Jan Zuiderveld, and Sandro Pezzelle. "Beyond divergent creativity: A human-based evaluation of creativity in large language models." Findings of the Association for Computational Linguistics: EACL 2026. 2026.
  • [5] Schapiro, S., Gladstone, A., Black, J., & Ji, H. (2026). Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers. arXiv preprint arXiv:2605.13450.
  • [6] Varshney, L. R., Rajani, N. F., & Socher, R. (2020). Explaining creative artifacts. arXiv preprint arXiv:2010.07126.
  • [7] Johnson, D. R., Kaufman, J. C., Baker, B. S., Patterson, J. D., Barbot, B., Green, A. E., ... & Beaty, R. E. (2023). Divergent semantic integration (DSI): Extracting creativity from narratives with distributional semantic modeling. Behavior research methods, 55(7), 3726-3759.
  • [8] Varshney, L. R. (2019). Mathematical limit theorems for computational creativity. IBM Journal of Research and Development, 63(1), 2-1.
  • [9] Gray, Kurt, Stephen Anderson, Eric Evan Chen, John Michael Kelly, Michael S. Christian, John Patrick, Laura Huang, Yoed N. Kenett, and Kevin Lewis. "“Forward flow”: A new measure to quantify free thought and predict creativity." American Psychologist 74, no. 5 (2019): 539.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.