- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces Universe of Thoughts (UoT) framework by operationalizing Boden's three modes of creativity, namely, combinational, exploratory and transformational creativity into three LLM search procedures: C-UoT, E-UoT and T-UoT. C-UoT combines solutions through analogy, E-UoT explores the search space for unexplored solutions and T-UoT transforms the underlying rules of the problem to find new solutions.
For evaluation, the authors consider a setup characterized by two properties: degree of constraint and objective specificity. As this allows a 2x2 task regime, they choose at least one task per quadrant where tasks for two quadrants are chosen from the HypoArena benchmark and tasks for the other two are introduced by the authors. They compare their method to other reasoning strategies such as CoT, ToT, and EGoT using some open-source and closed-source models. They score generations using both humans and LLMs. They show that different reasoning methods perform best on those tasks that are more suitable for it in how it is operationalized with respect to the 2x2 task regime.
Critical strengths
[S1] The paper is well-written, clear and appropriately structured.
[S2] The proposed framework is theoretically grounded in computational creativity research.
[S3] The authors evaluate their methods on both their proposed tasks and also on an external benchmark and compare to both frontier LLMs and other similar reasoning methods.
Critical weaknesses
[W1] There seems to be some coupling and task-specific behaviour between the different search procedures and it is not clear why this is the case. T-UoT algorithm uses a CREATIVESEARCH procedure once a transformation is applied, but this procedure is instantiated to E-UoT in the novel creative reasoning tasks and C-UoT in the HypoArena task. This is in contrast to the main claim that the three modes are distinct and it is not clear whether T-UoT's success on the novel reasoning tasks can be attributed to the transformation alone.
[W2] Some of the reported creativity scores are close to each other and there are no confidence intervals, significance tests or inter-rater reliability results among the ten human raters.
[W3] The separation of the utility and feasibility is not well supported. Utility or usefulness aspect of creativity generally covers all those dimensions of value for an artifact. Implausible or unqualified solution can not really be useful, so these dimensions are coupled.
[W4] The corpus of known solutions is too small (3-4 solutions per problem) to provide a meaningful novelty measure.
[W5] While the 2x2 task regime seems to nicely map onto the types of creativity, it is not clear how those tasks satisfy different degrees of constraints and objective specificity? How do we determine that Society task is low constraint and low specificity and Bridge and Electricity are high-specificity? Is it even possible to quantify these dimensions?
[W6] The novelty of the introduced reasoning tasks is questionable. These are just a few problems that can be at most part of a broader Design Solutions Task (Distefano et al. 2025).
[W7] While the work is theoretically grounded in the computational creativity literature (Boden's framework), it does not discuss any other works that have (at least theoretically) operationalized these types of creativity. A notable one is Geraint A. Wiggins 2006, "A preliminary framework for description, analysisand comparison of creative systems". I think the paper should discuss how their framework relates to or differs from this work.
Questions for authors
See weaknesses above.