acceptodds
Under review as a conference paper at ICLR 2027

TriCreativity: Benchmarking and Eliciting Combinational, Exploratory, and Transformational Creativity for Large Language Models

Abstract

Creativity is an important capability for large language models in scientific discovery, technological innovation, and open-ended problem solving. Existing benchmarks span multiple domains, but leave gaps in distinguishing the creative demands of tasks and assessing departures from existing conceptual frameworks. To this end, this paper introduces TriCreativity, a benchmark combining Boden’s combinational, exploratory, and transformational taxonomy with multidimensional evaluation. We also introduce Counterfactual Transformation (CFT), a dataset that supplies counterfactual changes to fundamental premises. Models must develop additional representations and organizing principles to construct coherent and useful conceptual frameworks under the changed premises. Across 11 large language models spanning multiple families and sizes, creative strengths vary by creative demand and domain: even within exploratory creativity, mathematics, coding, and writing favor different models. Within-family scaling yields uneven gains. Most evaluated generic reasoning procedures improve selected tasks or metrics but struggle to achieve joint gains across creative objectives. Thus, we further introduce Taxonomy-Guided Creativity Elicitation (TCE), a training-free method that coordinates generation, search, selection, and verification around task-specific creative requirements and achieves the best aggregate performance among the compared procedures. Novelty and appropriateness are strongly associated in CFT but weakly related in constrained writing. Together with output-level cases, this suggests that when familiar premises fail, effective problem solving may depend not only on finding alternative answers but on constructing new representations and organizing principles.

Primary area:
datasets and benchmarks
Keywords:
large language models, creativity evaluation, creativity benchmark, transformational creativity, multidimensional evaluation, creativity elicitation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vbZYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TriCreativity, a benchmark for evaluating LLM creativity based on Boden’s three categories: combinational, exploratory, and transformational creativity. The authors organize existing benchmarks according to their dominant creative demands and introduce Counterfactual Transformation (CFT), a new benchmark designed to evaluate transformational creativity through changes to fundamental premises. They also propose Taxonomy-Guided Creativity Elicitation (TCE), a training-free framework that selects and coordinates reasoning operators according to task-specific creative demands.

The authors evaluate 11 LLMs across eight tasks and investigate differences in creative capabilities across models, tasks, and evaluation dimensions. They also compare TCE against generic reasoning methods through ablation studies and inference-budget-controlled experiments.

Critical strengths
  1. Comprehensive empirical evaluation. The paper evaluates 11 models across diverse tasks and creativity dimensions. The experiments include human evaluation on a subset of outputs, statistical analysis, ablation studies, and inference-budget-controlled comparisons, which strengthen the empirical investigation.

  2. Novel benchmark for transformational creativity. CFT is an interesting contribution that evaluates models’ ability to construct coherent and useful conceptual frameworks under counterfactual changes to fundamental premises. This provides a concrete approach to evaluating an aspect of transformational creativity that is challenging to measure.

  3. Useful insights into task-dependent creativity. The results demonstrate that model performance varies substantially across creative demands and domains, suggesting that overall benchmark scores can obscure important differences in creative capabilities.

Critical weaknesses
  1. Insufficient validation of the task-to-creativity taxonomy.

The paper categorizes each task according to its dominant creative demand. While this is a reasonable design choice, the justification for these assignments could be strengthened. In particular, open-ended tasks may involve multiple types of creativity depending on the reasoning process employed.

For example, an open-ended problem-solving task could involve combining concepts, exploring existing solution spaces, or transforming underlying assumptions. Although the paper acknowledges that tasks may involve multiple creative abilities, it is unclear how the authors validate that the assigned category represents the dominant creative demand.

This is particularly important because one of the paper’s main contributions is to compare creative capabilities across Boden’s taxonomy.

  1. Data contamination and benchmark exposure.

Several included benchmarks are publicly available, raising concerns about potential contamination of frontier models’ training data. This is particularly relevant to tasks with canonical answers, such as BATS and RAT, where memorization may contribute to performance. Although the authors acknowledge benchmark exposure as a limitation, the paper does not appear to provide a systematic contamination analysis. Additional checks, such as paraphrased or newly generated evaluation items, would help clarify whether the reported results reflect creative reasoning rather than familiarity with existing benchmark examples.

  1. Unclear methodological novelty of TCE relative to existing creativity frameworks.

TCE coordinates prompt-based reasoning operators based on task-specific creativity requirements. However, previous work, including “Universe of Thoughts”, has already introduced reasoning frameworks inspired by Boden’s three creativity types, including combinational, exploratory, and transformational operations.

Although TCE introduces task-adaptive operator selection and coordinated execution, the distinction between its methodological contribution and existing creativity-oriented reasoning frameworks is unclear.

In particular, an experimental comparison with an existing taxonomy-based framework, or a more explicit analysis of the new components responsible for TCE’s improvements, would strengthen the novelty claim.

  1. Limited consistency of TCE’s task-level improvements.

While TCE achieves the highest aggregate performance, it does not consistently outperform other methods on individual tasks. For example, Table 3 shows that TCE achieves the highest task-level composite score only on NeoCoder, despite using creativity-specific operator pools. The call-budget-controlled experiments in Table 4 provide stronger evidence of TCE’s effectiveness, but the differences between the two comparisons and the sources of task-specific improvements deserve further discussion.

Clarifying why taxonomy-guided elicitation is effective for some tasks but less competitive for others would strengthen the central argument that different creativity types benefit from specialized reasoning operations.

Questions for authors
  1. Task categorization: What criteria were used to determine the dominant creativity type of each task? Have the authors considered validating these assignments through expert annotation or evaluating tasks under multiple creativity categories?

  2. Data contamination: Could the authors provide additional evidence that model performance on publicly available benchmarks is not substantially influenced by training-data exposure?

  3. Novelty of TCE: What is the key methodological distinction between TCE and existing creativity-oriented reasoning frameworks, particularly Universe of Thoughts? Which components represent fundamentally new contributions rather than task-specific compositions of existing reasoning operators?

  4. TCE effectiveness: Why does TCE achieve the highest overall score but fail to consistently outperform generic methods on individual tasks in Table 3? How should these results be interpreted in relation to the claimed benefits of creativity-specific operator selection?

Reviewer_rrofOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work introduces TriCreativity, a benchmark for creativity which groups eight creativity tasks by Boden’s combinational, exploratory, and transformational types.

Across 11 models, they find that claude leads overall while gemini leads combinational creativity. The proposed method TCE beats generic prompting methods.

But overall, the paper is hard to follow.

Critical strengths

None

Critical weaknesses

CW1: The paper's presentation is really underwhelming. Many jargon are ill defined and not properly introduced. CW2: The paper is clearly AI written. CW3: It is unclear whether this paper probes an aspect different from other creativity benchmarks.

Questions for authors

None.

Reviewer_5bT3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TriCreativity, a benchmark combining ideas, searching under fixed rules, and designing systems under changed rules. Its new Counterfactual Transformation (CFT) task provides an existing system, changes to its rules, and goals as inputs; the model must propose a design that satisfies them. The author also introduce Taxonomy-Guided Creativity Elicitation (TCE), a prompting method that coordinates generation, search, selection, and verification around task-specific creative requirements. Using the same underlying model, it raises the aggregate benchmark score from Self-Refine’s 0.572 to 0.603. On CFT, higher novelty and appropriateness accompany lower fluency and flexibility, leaving the composite score slightly below direct prompting. The author conclude that effective problem solving may depend not only on finding alternative answers but on constructing new representations and organizing principles.

Critical strengths

The proposed new counterfactual transformation task is interesting, asking a model to redesign a system under supplied rule changes.

The experiment and evaluation setup is sound, and multiple metrics are reported. (e.g. TCE improves CFT novelty and appropriateness but reduces fluency and flexibility. In code, constraint compliance is reported separately from executable correctness)

Critical weaknesses

The design of counterfactual transformation task might not be solid. A high score may reflect systems design skill or use of familiar cues rather than a distinct general creative ability.

The baselines compared against seems to be limited and on a weaker side.

Questions for authors

Would a fixed workflow perform just as well as adaptive selection? This would separate how useful adaptation is from the operators themselves.

Would the gains survive token-matched and stronger open-ended baselines?

1 comment

Dear Reviewer 5bT3,

Thank you again for your focused questions. Following your suggestions, our response above compares TCE with a fixed workflow and adds SELF-DISCOVER and Universe of Thoughts under the same call budget.

We would be happy to provide further details on any of these results before the discussion period ends.

Reviewer_S5eBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TriCreativity, a benchmark organizing eight tasks into combinational, exploratory, and transformational creativity. It proposes Counterfactual Transformation (CFT), containing 1,308 items across six domains, and Taxonomy-Guided Creativity Elicitation (TCE), a training-free prompting framework. Experiments on 11 language models reveal task-dependent performance differences. TCE achieves the highest aggregate score among the evaluated elicitation methods, including under inference-budget controls.

Critical strengths
  1. CFT introduces an original task formulation for evaluating model-generated conceptual frameworks under counterfactual rule changes.
  2. The evaluation covers multiple models and creativity dimensions, with ablation studies, inference-budget controls, and human validation of automated scoring.
Critical weaknesses
  1. Insufficient construct validity. CFT evaluates framework reconstruction under supplied rule changes, but the current validation does not establish whether its scores reliably distinguish framework-level transformation from routine adaptation. This limits the evidence supporting its central measurement claim.
  2. Limited demonstrated discriminative ability. CFT shows the narrowest cross-model separation among the eight tasks, while most rubric scores cluster within a relatively narrow middle range. It remains unclear whether this reflects genuinely similar model capabilities or limited evaluator sensitivity.
  3. Limited evaluation reliability. Human–LLM agreement on CFT is only moderate and becomes weaker after controlling for response length. The novelty–appropriateness association is also substantially lower under cross-judge than same-judge scoring, raising concerns about judge-specific effects and the interpretation of this correlation.
  4. Limited methodological novelty. TCE combines task-specific prompting operators, routing, selection, and verification, with substantial overlap with existing modular reasoning approaches such as SELF-DISCOVER [1] and Universe of Thoughts [2]. Direct comparisons with these closely related methods are not reported, leaving the incremental contribution unclear.
  5. Limited gains on the proposed transformational task. On CFT, TCE achieves essentially the same aggregate score as direct prompting (0.626 vs. 0.628), despite improving selected dimensions. This weakens the evidence that taxonomy-guided elicitation provides an overall advantage for transformational creativity.

[1] SELF-DISCOVER: Large Language Models Self-Compose Reasoning Structures (2024), NeurIPS.

[2] Universe of Thoughts: Enabling Creative Reasoning with Large Language Models (2025), arXiv.

Questions for authors
  1. How is CFT validated to distinguish framework-level transformation from routine adaptation?
  2. Does the compressed CFT score distribution reflect limited model capabilities or limited evaluator sensitivity?
  3. How robust are CFT model rankings and TCE improvements under independent human evaluation?

If the authors can address these concerns, I would be open to increasing my score.

1 comment

Dear Reviewer S5eB,

Thank you again for taking the time to review our paper. Our response above answers each of your three questions with a new experiment, and adds the comparison with SELF-DISCOVER and Universe of Thoughts you suggested.

If you have any remaining questions or concerns, we would be grateful for the opportunity to address them before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.