- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Creative AI Researcher (CAR), an agentic framework for scientific research ideation that combines literature grounding, iterative development, and explicit creative operators. CAR constructs a semantic knowledge graph from retrieved papers and supports four graph-editing operations: bisociation, analogy, exploration, and elaboration. The operations are subject to structural validation, and their execution records provide an inspectable trace of idea development.
The authors evaluate CAR on 40 LiveIdeaBench topics using two LLM backbones, as well as FML-bench-Lite and BioXArena with a shared experiment executor. CAR achieves significantly higher methodological novelty (MN) than single-shot prompting and The AI Scientist, but its differences from RIA and The AI Scientist-v2 are not statistically significant. It also obtains the highest pairwise win rates among the evaluated AI methods on both execution benchmarks. However, the ablation study does not establish a consistent quality improvement from explicit creative operators beyond free-form iteration.
Critical strengths
The paper provides a concrete implementation of creativity-inspired operations as validated graph edits. This makes the intermediate steps of research ideation more explicit than in approaches relying primarily on natural-language generation and refinement. The separation of literature grounding, iteration, and operator validation also enables a more systematic analysis of the ideation process. I consider the inspectability of these operations a useful contribution, even though the traces do not establish causal explanations.
The evaluation considers both proposal-level quality and downstream experimental performance. In particular, using the same executor for CAR and The AI Scientist-v2 provides a meaningful comparison of their generated ideas. The results on FML-bench-Lite and BioXArena offer encouraging evidence of practical utility. I also appreciate that the authors report multiple-comparison corrections and explicitly acknowledge that the ablations do not establish a consistent quality benefit from the creative operators.
Critical weaknesses
The four creative operators are a central methodological component, but the ablation results in Table 5 provide limited evidence that they improve idea quality beyond ordinary iterative reasoning. Compared with Freeform, which retains iterative development and graph tools without explicit operators, CAR has an MN difference of -0.02 ± 0.12 on GPT-5.4 and +0.37 ± 0.12 on GLM-5.2. Neither difference survives Holm correction. Removing individual operators, disabling validation, or shuffling relation labels also fails to produce a consistent, statistically significant MN reduction across both backbones.
The authors acknowledge this limitation. Nevertheless, it leaves the main methodological question unresolved: whether the proposed operator-based decomposition contributes meaningfully to idea quality beyond literature grounding and iteration. The positive execution results do not resolve this question because the corresponding operator ablations are not evaluated on the execution benchmarks.
The traceability mechanism is potentially useful, but the reported validation primarily concerns the consistency of recorded operations and entity references. Appendix G shows that nearly all declared entity indices appear in preceding logs. However, this does not establish that the referenced literature substantively supports the generated proposal or that the accepted operations are scientifically meaningful.
Moreover, similarly high index-appearance coverage is reported for the audited ablation conditions. The paper therefore does not clearly demonstrate what additional practical value the structured operator traces provide over ordinary retrieval and interaction logs. A targeted semantic evaluation, even on a small sample of proposals, would strengthen this contribution. The concern is not that the authors overclaim causal faithfulness, which they explicitly avoid, but that the usefulness of the proposed trace representation remains largely untested.
Questions for authors
-
Is there an execution-level comparison between CAR and the Freeform ablation, using the same executor and comparable computational budgets? Such a comparison would help determine whether the downstream performance gains are attributable to the explicit creative operators rather than to literature grounding and iterative development more generally.
-
Can the authors provide evidence beyond index-appearance coverage that the structured traces accurately represent meaningful relationships between source literature, intermediate graph edits, and final proposals? In particular, a small semantic audit or comparison with ordinary agent logs would help clarify the distinctive value of the proposed traceability mechanism.
General Response to Reviewers (1/2)
We thank Reviewers rzo3, Pi9H and kUSM for their efforts and time put into their reviews and their constructive comments on our work. The three reviews converge on the same important points and also bring up other salient points.
Operators versus free-form iteration (rzo3 W1, Pi9H W2/Q1, kUSM W1). We agree, and the paper states that the results do not establish a consistent quality benefit from the creative operators beyond free-form iteration (−0.02 ± 0.12 MN on GPT-5.4, +0.37 ± 0.12 on GLM-5.2, neither surviving correction). Removing single operators, disabling operator validation and shuffling relation labels also produced no consistent MN reduction across backbones. All three reviews describe the operators as a core methodological component of the paper, and we accept that a core methodological component without a demonstrated benefit weakens it. The only property we can currently claim for the operators is a typed and validated record of each creative transformation, and point 4 discusses whether that record is useful. We will reduce the prominence of the operators as a contribution and make the absence of a demonstrated benefit a main finding. The next version will be organized around the question of whether operators drawn from theories of creativity improve research ideation by LLMs, answered by decomposing the pipeline into its components.
What the gain is attributable to (kUSM W1). The reviewer is right that our ablations do not isolate grounding. On 12 of the 40 LiveIdeaBench keywords, we ran free-form iteration with the literature graph removed. MN did not fall relative to the graph-grounded Freeform arm (−0.03 ± 0.27 on GPT-5.4, +0.61 ± 0.26 on GLM-5.2). This suggests that the gain over single-shot prompting comes from the agent pipeline as a whole and not specifically from the retrieved literature. The paper treats grounding as a shared setting rather than testing it, and we will state that its contribution is not isolated.
Execution-level comparison with Freeform (kUSM Q1). We ran Freeform ideas through the same executor on the nine BioXArena tasks, one run per task. On the three tasks left unchanged in the benchmark organizers’ corrected data release of 7 October 2026, Freeform scores 0.861, 0.631 and 0.771, and CAR scores 0.897, 0.624 and 0.775 (cyp-inhibition macro-AUC, breast-cancer macro-F1, pan-cancer survival c-index). The organizers rebuilt the other six tasks in that release to correct leakage, label and split problems, and we will report execution results on the corrected release in the revision.
Usefulness of traces (kUSM W2/Q2, rzo3 W2). We agree. The paper limits the trace claim to procedural provenance and states that structural validity does not establish scientific correctness. Declared-entity coverage is 99.96% to 100% in the ablation arms, at least as high as CAR’s 99.70% to 100%, so it does not distinguish operator traces from ordinary logs. We will add a blinded human audit comparing operator traces with free-form logs on a sample of ideas in the next version of the paper.