- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TriCreativity, a benchmark for evaluating LLM creativity based on Boden’s three categories: combinational, exploratory, and transformational creativity. The authors organize existing benchmarks according to their dominant creative demands and introduce Counterfactual Transformation (CFT), a new benchmark designed to evaluate transformational creativity through changes to fundamental premises. They also propose Taxonomy-Guided Creativity Elicitation (TCE), a training-free framework that selects and coordinates reasoning operators according to task-specific creative demands.
The authors evaluate 11 LLMs across eight tasks and investigate differences in creative capabilities across models, tasks, and evaluation dimensions. They also compare TCE against generic reasoning methods through ablation studies and inference-budget-controlled experiments.
Critical strengths
-
Comprehensive empirical evaluation. The paper evaluates 11 models across diverse tasks and creativity dimensions. The experiments include human evaluation on a subset of outputs, statistical analysis, ablation studies, and inference-budget-controlled comparisons, which strengthen the empirical investigation.
-
Novel benchmark for transformational creativity. CFT is an interesting contribution that evaluates models’ ability to construct coherent and useful conceptual frameworks under counterfactual changes to fundamental premises. This provides a concrete approach to evaluating an aspect of transformational creativity that is challenging to measure.
-
Useful insights into task-dependent creativity. The results demonstrate that model performance varies substantially across creative demands and domains, suggesting that overall benchmark scores can obscure important differences in creative capabilities.
Critical weaknesses
- Insufficient validation of the task-to-creativity taxonomy.
The paper categorizes each task according to its dominant creative demand. While this is a reasonable design choice, the justification for these assignments could be strengthened. In particular, open-ended tasks may involve multiple types of creativity depending on the reasoning process employed.
For example, an open-ended problem-solving task could involve combining concepts, exploring existing solution spaces, or transforming underlying assumptions. Although the paper acknowledges that tasks may involve multiple creative abilities, it is unclear how the authors validate that the assigned category represents the dominant creative demand.
This is particularly important because one of the paper’s main contributions is to compare creative capabilities across Boden’s taxonomy.
- Data contamination and benchmark exposure.
Several included benchmarks are publicly available, raising concerns about potential contamination of frontier models’ training data. This is particularly relevant to tasks with canonical answers, such as BATS and RAT, where memorization may contribute to performance. Although the authors acknowledge benchmark exposure as a limitation, the paper does not appear to provide a systematic contamination analysis. Additional checks, such as paraphrased or newly generated evaluation items, would help clarify whether the reported results reflect creative reasoning rather than familiarity with existing benchmark examples.
- Unclear methodological novelty of TCE relative to existing creativity frameworks.
TCE coordinates prompt-based reasoning operators based on task-specific creativity requirements. However, previous work, including “Universe of Thoughts”, has already introduced reasoning frameworks inspired by Boden’s three creativity types, including combinational, exploratory, and transformational operations.
Although TCE introduces task-adaptive operator selection and coordinated execution, the distinction between its methodological contribution and existing creativity-oriented reasoning frameworks is unclear.
In particular, an experimental comparison with an existing taxonomy-based framework, or a more explicit analysis of the new components responsible for TCE’s improvements, would strengthen the novelty claim.
- Limited consistency of TCE’s task-level improvements.
While TCE achieves the highest aggregate performance, it does not consistently outperform other methods on individual tasks. For example, Table 3 shows that TCE achieves the highest task-level composite score only on NeoCoder, despite using creativity-specific operator pools. The call-budget-controlled experiments in Table 4 provide stronger evidence of TCE’s effectiveness, but the differences between the two comparisons and the sources of task-specific improvements deserve further discussion.
Clarifying why taxonomy-guided elicitation is effective for some tasks but less competitive for others would strengthen the central argument that different creativity types benefit from specialized reasoning operations.
Questions for authors
-
Task categorization: What criteria were used to determine the dominant creativity type of each task? Have the authors considered validating these assignments through expert annotation or evaluating tasks under multiple creativity categories?
-
Data contamination: Could the authors provide additional evidence that model performance on publicly available benchmarks is not substantially influenced by training-data exposure?
-
Novelty of TCE: What is the key methodological distinction between TCE and existing creativity-oriented reasoning frameworks, particularly Universe of Thoughts? Which components represent fundamentally new contributions rather than task-specific compositions of existing reasoning operators?
-
TCE effectiveness: Why does TCE achieve the highest overall score but fail to consistently outperform generic methods on individual tasks in Table 3? How should these results be interpreted in relation to the claimed benefits of creativity-specific operator selection?
Follow-up
Dear Reviewer 5bT3,
Thank you again for your focused questions. Following your suggestions, our response above compares TCE with a fixed workflow and adds SELF-DISCOVER and Universe of Thoughts under the same call budget.
We would be happy to provide further details on any of these results before the discussion period ends.