- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
- introduces KOMBINE benchmark for comparing combinatorial capability of llms
- takeaway: LLM show intutive homogenity, found new analogies, and reveal creative strengths in LLMs' capacity for association and analogy
- scoring on useful, surpirsing and original
- blending hardest task
Critical strengths
- introduces a new benchmark
- large scale study on LLMs
- tool to measure creativeness
- 3 domains, association, analogyy, blending
Critical weaknesses
-
how do you differentiate creativeness against factual incorrectness, truthfulness?
-
score is driven by validity: if not valid then 0, which doesnt aligns to interpret creativity
-
absrtaction bottleneck may come from judge end, which hasnt been explored
-
correlation of judge to human connection is thin
-
missing related works
-
Creativity tests on LLMs: Stevenson et al. (2022) on GPT-3 and the Alternate Uses Task; Koivisto & Grassini (2023), where AI beat average humans but not the best ones; Guzik et al. (2023) on the Torrance tests; Organisciak et al. (2023) on automated creativity scoring.
-
Judging creative quality: Chakrabarty et al. (CHI 2024), "Art or Artifice?" (TTCW); and Lu et al. (2024), "AI as Humanity's Salieri" (Creativity Index). The second measures novelty against human text on the web, which directly addresses this paper's weakest point.
-
AIs sounding alike: Doshi & Hauser (2024, Science Advances) on AI raising individual creativity but reducing diversity across people; Padmakumar & He (ICLR 2024) on writing with LLMs reducing diversity; Anderson et al. (2024) on homogenisation; Kirk et al. (2024) on RLHF reducing output diversity.
Questions for authors
2 comments
Author Rebuttal [2/2]
Weakness 3 (Is the abstraction bottleneck a robust finding?)
Multiple other reviewers suggested causally testing the abstraction bottleneck, a key finding which was previously only supported with correlational claims.
To do this, we took six models (claude-opus-5, gpt-5.2, deepseek-chat-v3-0324, llama-3.3-70b, qwen-2.5-72b, gpt-4o-mini) and assessed how well they could elaborate a coherent blended concept on items on which they previously failed by giving them valid generic spaces that other (often stronger) models discovered on those same items.
| Model | Baseline | Given Abstraction | Abstraction Bottleneck Gap |
|---|---|---|---|
| claude-opus-5 | 48.3 | 86.2 | 37.9pp |
| gpt-5.2 | 36.7 | 80.0 | 43.3pp |
| deepseek-chat-v3-0324 | 23.3 | 83.3 | 60.0pp |
| llama-3.3-70b | 30.0 | 63.3 | 33.3pp |
| qwen-2.5-72b | 26.7 | 60.0 | 33.3pp |
| gpt-4o-mini | 16.7 | 53.3 | 36.6pp |
| Average | 30.3±4.5 | 71.0±5.5 | 40.7±4.1 pp |
Results: On blends a model previously failed, supplying a valid generic space from another model improves the rate at which models are able to elaborate a coherent blended space by 40.7 percentage points (pp) on average. For example, when asked to blend “Democracy” and “Banking,” gpt-4o-mini’s default generic space was “a financial system that allocates resources based on collective input,” which only applies to banking. However, when given a valid generic space (“a system converting personal endowments into collective claims through trusted agents”), gpt-4o-mini was then able to elaborate a coherent blended concept. In summary, the results from this experiment provide significant evidence that the abstraction bottleneck is a causal bottleneck to performance.
Weakness 4 (Validity of LLM Judges)
LLM judges agree with human raters as often as human raters agree with each other: We conducted a study where three of the authors independently and blindly rated 40 answers from 12 entity pairs to assess all judge-based scoring dimensions used on Kombine. This allowed us to assess how often two humans agree with each other (human-human agreement), versus how often the LLM judges' verdict (from majority vote) agrees with each human annotator, averaged over annotators (LLM–human agreement).
| Dimension | Human–human agreement | LLM–human agreement |
|---|---|---|
| Analogy: valid analogy | 69% | 72% |
| Analogy: mapping used | 44% | 67% |
| Analogy: coherence | 56% | 74% |
| Blending: generic space | 56% | 58% |
| Blending: coherent | 77% | 86% |
| Blending: scope | 44% | 43% |
Results. On five of six subjectively scored dimensions, LLM-human agreement is better than human-human agreement. On analogy (validity, mapping used, coherence), as well as blending (generic space, blend coherence), the panel agrees with each rater at least as often as the other raters do. The one dimension where LLM-human agreement is slightly worse than human-human agreement is blending semantic integration quality, but LLM-human agreement is only 1 percentage point worse than human-human agreement here. Overall the LLM judge verdict on Kombine is as reliable as a single human annotator.
Weakness 5 (Missing related works)
Thank you for pointing these out! We will add them to the final draft.
References
- [1] Runco, M. A., & Jaeger, G. J. (2012). The standard definition of creativity. Creativity research journal, 24(1), 92-96.
- [2] Maher, Mary Lou. "Evaluating creativity in humans, computers, and collectively intelligent systems." Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design. 2010.
- [3] Nakajima, Kumiko, Jan Zuiderveld, and Sandro Pezzelle. "Beyond divergent creativity: A human-based evaluation of creativity in large language models." Findings of the Association for Computational Linguistics: EACL 2026. 2026.
- [4] Schapiro, S., Gladstone, A., Black, J., & Ji, H. (2026). Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontiers. arXiv preprint arXiv:2605.13450.
- [5] Liu, J., Min, S., Zettlemoyer, L., Choi, Y., & Hajishirzi, H. (2024). Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. arXiv preprint arXiv:2401.17377.
Author Rebuttal [1/2]
We sincerely thank the reviewer for the careful analysis of our work.
Weakness 1 (Differentiating creativity from factual incorrectness or truthfulness)
To answer this question and another related question poised by Reviewer En17, we conducted an analysis to distinguish between outputs obtained from genuinely creative reasoning versus outputs obtained by models regurgitating knowledge from their training data. To search for memorized inventions, we query the 4.6 trillion token OLMo2 pre-training mix from Infini-gram [5]. We define an (f,i,p)-memorized invention as a concept whose name (e.g. “sun loaf”) occurs verbatim at least f times in this pre-training corpora, whose name occurs within i tokens of at least one of the two inputs to an item (e.g., “photosynthesis” and “bread”), and for which at least p of the invention’s properties also occur within ‘i’ tokens of the concept’s name.
Results. By these criteria, and for (f,i,p)=(10,100,2), 43 of the 2,070 inventions (2.1%) are memorized. Unlike inventive multiples, models on average produce memorized concepts on analogy and blending at essentially equal rates (2.1% vs 2.0%). Furthermore, as shown above, the models which produce memorized inventions at the highest rate mostly come from the Meta, Microsoft, and Qwen families, with llama-3.3-70b at 17% (10 of 58), claude-sonnet-4.5 at 8% (5 of 60), qwen-2.5-72b at 7%, phi-4 at 6%, and llama-4-maverick at 5%. Conversely, 15 of 35 models never produce one. They include the current frontier: opus-5, fable-5, sonnet-5, sonnet-4.6, gpt-5.6-sol, gpt-5.2, gemini-3.1-pro, gemini-2.5-pro, deepseek-r1, and both Grok and both GLM models.
Overall, this analysis suggests that smaller, open-weight models are more likely to produce memorized concepts that rely on previously encountered relationships (and rely on factuality rather than creativity) than larger frontier models. We plan to incorporate this analysis into our revisions and want to thank you for raising this question.
Weakness 2 (Is Kombine robust to how scoring dimensions are aggregated?)
The utility gate operationalizes the standard two-criterion definition of a creative product, which must be novel and useful [1]. This follows a precedent in the creative psychometrics literature dating back to Maher (2010) which defines utility in terms of task-specific constraints [2]. Indeed, recent creativity tests for LLMs like the CDAT [3] and DRAT [4] have (appropriately) defined overall scoring as novelty conditioned on utility, following the precedent set by [2]. However, to assess the robustness of the benchmark to various other ways of aggregating task scores, we report the Spearman rank correlation between the default benchmark scoring and two alternative variants, as defined below.
Results. Under two different aggregations (removing the utility gate on each item, or taking the geometric mean of each of the three task scores to compute the overall score), the spearman rank correlation with the default scoring remains very high (0.965 at worst). This indicates the ranking is robust to how scoring dimensions are aggregated, as well as whether utility is treated as a gate or as a separate dimension.