acceptodds
Under review as a conference paper at ICLR 2027

Ground, Iterate, Trace: Decomposing AI Research Ideation

Abstract

Large language models have been widely used to generate plausible research ideas, yet it remains unclear what makes automated ideation effective and how generated novel proposals arise from prior scientific work. We introduce the Creative AI Researcher (CAR), an agentic framework combining literature grounding, iterative development and explicit conceptual operators. CAR constructs a semantic knowledge graph from retrieved literature and equips an ideation agent with graph-retrieval tools and four operators: bisociation, analogy, exploration and elaboration. Each operator call is checked against structural acceptance conditions and recorded, producing a trace of intermediate transformations and declared literature sources. We evaluate CAR with two language-model backbones across 40 LiveIdeaBench research topics and two execution benchmarks. CAR achieves significantly higher methodological novelty (MN) than single-shot prompting and The AI Scientist. Its methodological novelty does not differ significantly from the Research Ideation Agent, while its feasibility scores are higher on both backbones. With a shared experiment executor, CAR achieves the highest pairwise win rate among the evaluated AI methods on FML-bench-Lite and BioXArena. In a ten-condition ablation study, MN point estimates favor CAR over one-step proposal on both backbones, but no MN contrast survives correction for multiple comparisons. The results do not establish a consistent quality benefit from explicit operators beyond free-form iteration. CAR makes the sequence of operations and declared literature sources inspectable, separating judged idea quality from procedural traceability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI for science, scientific ideation, computational creativity, large language models, knowledge graphs, agentic reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kUSMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Creative AI Researcher (CAR), an agentic framework for scientific research ideation that combines literature grounding, iterative development, and explicit creative operators. CAR constructs a semantic knowledge graph from retrieved papers and supports four graph-editing operations: bisociation, analogy, exploration, and elaboration. The operations are subject to structural validation, and their execution records provide an inspectable trace of idea development.

The authors evaluate CAR on 40 LiveIdeaBench topics using two LLM backbones, as well as FML-bench-Lite and BioXArena with a shared experiment executor. CAR achieves significantly higher methodological novelty (MN) than single-shot prompting and The AI Scientist, but its differences from RIA and The AI Scientist-v2 are not statistically significant. It also obtains the highest pairwise win rates among the evaluated AI methods on both execution benchmarks. However, the ablation study does not establish a consistent quality improvement from explicit creative operators beyond free-form iteration.

Critical strengths

The paper provides a concrete implementation of creativity-inspired operations as validated graph edits. This makes the intermediate steps of research ideation more explicit than in approaches relying primarily on natural-language generation and refinement. The separation of literature grounding, iteration, and operator validation also enables a more systematic analysis of the ideation process. I consider the inspectability of these operations a useful contribution, even though the traces do not establish causal explanations.

The evaluation considers both proposal-level quality and downstream experimental performance. In particular, using the same executor for CAR and The AI Scientist-v2 provides a meaningful comparison of their generated ideas. The results on FML-bench-Lite and BioXArena offer encouraging evidence of practical utility. I also appreciate that the authors report multiple-comparison corrections and explicitly acknowledge that the ablations do not establish a consistent quality benefit from the creative operators.

Critical weaknesses

The four creative operators are a central methodological component, but the ablation results in Table 5 provide limited evidence that they improve idea quality beyond ordinary iterative reasoning. Compared with Freeform, which retains iterative development and graph tools without explicit operators, CAR has an MN difference of -0.02 ± 0.12 on GPT-5.4 and +0.37 ± 0.12 on GLM-5.2. Neither difference survives Holm correction. Removing individual operators, disabling validation, or shuffling relation labels also fails to produce a consistent, statistically significant MN reduction across both backbones.

The authors acknowledge this limitation. Nevertheless, it leaves the main methodological question unresolved: whether the proposed operator-based decomposition contributes meaningfully to idea quality beyond literature grounding and iteration. The positive execution results do not resolve this question because the corresponding operator ablations are not evaluated on the execution benchmarks.

The traceability mechanism is potentially useful, but the reported validation primarily concerns the consistency of recorded operations and entity references. Appendix G shows that nearly all declared entity indices appear in preceding logs. However, this does not establish that the referenced literature substantively supports the generated proposal or that the accepted operations are scientifically meaningful.

Moreover, similarly high index-appearance coverage is reported for the audited ablation conditions. The paper therefore does not clearly demonstrate what additional practical value the structured operator traces provide over ordinary retrieval and interaction logs. A targeted semantic evaluation, even on a small sample of proposals, would strengthen this contribution. The concern is not that the authors overclaim causal faithfulness, which they explicitly avoid, but that the usefulness of the proposed trace representation remains largely untested.

Questions for authors
  • Is there an execution-level comparison between CAR and the Freeform ablation, using the same executor and comparable computational budgets? Such a comparison would help determine whether the downstream performance gains are attributable to the explicit creative operators rather than to literature grounding and iterative development more generally.

  • Can the authors provide evidence beyond index-appearance coverage that the structured traces accurately represent meaningful relationships between source literature, intermediate graph edits, and final proposals? In particular, a small semantic audit or comparison with ordinary agent logs would help clarify the distinctive value of the proposed traceability mechanism.

Reviewer_Pi9HOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Creative AI Researcher (CAR), an agent framework for automated scientific research ideation that combines a literature-based knowledge graph, iterative ideation, and four explicit creative operations: bisociation, analogy, exploration, and elaboration. CAR validates and records these operations through structural constraints, enabling generated research proposals to be traced back to relevant literature. The authors evaluate CAR on 40 research topics in LiveIdeaBench using two language models and further assess its execution performance on FML-bench-Lite and BioXArena. The results show that CAR outperforms single-prompt baselines and The AI Scientist in methodological novelty, achieving the highest pairwise win rates among the evaluated AI methods on both execution benchmarks. However, the ablation study does not establish that explicit creative operations provide significant additional benefits over free-form iterative ideation.

Critical strengths
  1. The paper addresses a practically relevant problem and presents a relatively comprehensive system design. The paper investigates how to generate research ideas that are grounded in the literature and traceable throughout the generation process. It integrates literature-based knowledge grounding, iterative ideation, and process tracing into a unified framework through the Ground, Iterate, and Trace stages. This design has practical value and offers a meaningful attempt to improve the transparency of AI-assisted research ideation.

  2. The evaluation covers multiple dimensions and includes downstream execution experiments. In addition to evaluating idea originality, feasibility, and diversity on LiveIdeaBench, the authors conduct downstream execution experiments on machine learning and biology-related tasks to investigate whether the generated ideas can be translated into executable methods. This provides more direct empirical evidence than relying solely on LLM-based assessments of idea quality. Although the current experiments remain limited in scale and evidential strength, the combination of idea quality evaluation and downstream execution validation is a worthwhile aspect of the work.

Critical weaknesses
  1. Limited novelty. Existing work has explored identifying research opportunities through the evolution of the literature (e.g., Chain of Idea) and leveraging large-scale scholarly literature and knowledge graphs for research ideation (e.g., SciPIP). Compared with these approaches, the substantive differences introduced by the Ground–Iterate–Trace framework and its four creativity operators remain unclear. The paper also fails to demonstrate that these design choices provide capabilities beyond those of existing methods, limiting its methodological novelty.

  2. Insufficient empirical evidence for the effectiveness of the core design. In the ablation study, the Freeform: iterative, no operators baseline achieves comparable Methodological Novelty scores to the full CAR system, with no statistically significant difference. This raises questions about whether the four creativity operators provide benefits beyond ordinary iterative generation. Furthermore, removing operator validation does not lead to a significant performance drop, further calling into question the necessity of these design choices. Overall, the paper does not sufficiently demonstrate that its core components contribute meaningfully to performance improvements.

  3. Limited feasibility of generated ideas and imperfect alignment between evaluation metrics and task objectives. CAR achieves relatively low Feasibility scores (4.97 for GPT-5.4 and 4.59 for GLM-5.2), suggesting that the practical feasibility of its generated ideas remains a concern. Meanwhile, its advantage in Fluency primarily reflects diversity among generated ideas rather than improvements in scientific value or practical utility. For research ideation, increased diversity alone is insufficient to establish the overall effectiveness of the proposed method, and the limited scale of the downstream execution experiments provides insufficient evidence of its practical research value.

  4. Limitations in the reliability of novelty evaluation. The paper uses an LLM-based evaluator to assess the overlap between proposed methods and existing work, but its correlation with expert ratings is relatively modest (Spearman’s ρ = 0.39). Moreover, the samples used to develop the evaluator do not constitute an independent held-out set. These limitations raise concerns about the reliability of the Methodological Novelty metric and weaken the evidence supporting the paper’s central claims.

Questions for authors
  1. Effectiveness of the Core Components: In the ablation study, the Freeform baseline achieves a method novelty score comparable to that of the full CAR framework. Could the authors further clarify the main advantages of the four creative operations over conventional iterative ideation? Do these operations yield more evident improvements on other evaluation metrics?
  2. Practical Feasibility of Generated Proposals: CAR achieves a relatively low Feasibility score. Could the authors elaborate on the practical feasibility of the generated proposals for real-world research, drawing on the downstream execution experiments? To what extent do the current evaluation results support the practical utility of CAR?
Reviewer_rzo3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the CAR, a agentic framework designed to generate research ideas through reasoning, iterative refinement. CAR begins by constructing an semantic knowledge graph from retrieved scientific literature, mapping out research questions, methodologies, findings, and their interconnections. Moreover, a ideation agent, which is a LLM, interacts with this knowledge graph using four structured creative operations: bisociation, analogy, exploration, and elaboration. Each operation is validated against predefined structural constraints and documented, ensuring the traceability of the ideation process. The authors evaluated CAR across 40 topics in LiveIdeaBench, as well as through execution benchmarks in FML-bench-Lite and BioXArena, with models such as GPT-5.4 and GLM-5.2. Experimental results demonstrate that this framework achieves greater innovation in research methodology compared to baselines.

Critical strengths

Experimental results demonstrate that this framework achieves greater innovation in research methodology compared to baselines.

Critical weaknesses

I believe the paper has several notable shortcomings:

  1. One of the core contributions of this work is the introduction of four structured creative operators designed to improve the scientific ideation process. However, the evidence provided by the ablation studies is limited and fails to sufficiently demonstrate that these operators outperform traditional iterative reasoning methods in enhancing the quality of ideation.
  2. The creative operators proposed are validated primarily through graph-structural constraints (such as entity connectivity, relational consistency, and the introduction of new entities). Yet, satisfying these constraints does not necessarily guarantee that the resulting conceptual transformations possess scientific significance, novelty, or logical validity. For instance, while connecting otherwise unrelated concepts via a "bridging entity" might satisfy the constraints for "bisociation," it does not guarantee the generation of a scientifically valuable hypothesis.
  3. CAR relies heavily on Large Language Models (LLMs) to extract scientific concepts and their relationships from a limited set of literature (at most 20 papers per research topic). Scientific literature, however, often contains complex methodological details, implicit dependencies, and nuanced research findings—elements that are difficult to accurately translate into discrete entities and relationships. Errors or omissions during literature retrieval, information extraction, or entity merging could significantly impact the creative operations performed by the ideation agent. The paper neither provides a systematic evaluation of the quality of the constructed knowledge graph nor explores how extraction errors or incomplete literature coverage might affect the subsequent ideation generation process.
Questions for authors

No

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers (1/2)

We thank Reviewers rzo3, Pi9H and kUSM for their efforts and time put into their reviews and their constructive comments on our work. The three reviews converge on the same important points and also bring up other salient points.

  1. Operators versus free-form iteration (rzo3 W1, Pi9H W2/Q1, kUSM W1). We agree, and the paper states that the results do not establish a consistent quality benefit from the creative operators beyond free-form iteration (−0.02 ± 0.12 MN on GPT-5.4, +0.37 ± 0.12 on GLM-5.2, neither surviving correction). Removing single operators, disabling operator validation and shuffling relation labels also produced no consistent MN reduction across backbones. All three reviews describe the operators as a core methodological component of the paper, and we accept that a core methodological component without a demonstrated benefit weakens it. The only property we can currently claim for the operators is a typed and validated record of each creative transformation, and point 4 discusses whether that record is useful. We will reduce the prominence of the operators as a contribution and make the absence of a demonstrated benefit a main finding. The next version will be organized around the question of whether operators drawn from theories of creativity improve research ideation by LLMs, answered by decomposing the pipeline into its components.

  2. What the gain is attributable to (kUSM W1). The reviewer is right that our ablations do not isolate grounding. On 12 of the 40 LiveIdeaBench keywords, we ran free-form iteration with the literature graph removed. MN did not fall relative to the graph-grounded Freeform arm (−0.03 ± 0.27 on GPT-5.4, +0.61 ± 0.26 on GLM-5.2). This suggests that the gain over single-shot prompting comes from the agent pipeline as a whole and not specifically from the retrieved literature. The paper treats grounding as a shared setting rather than testing it, and we will state that its contribution is not isolated.

  3. Execution-level comparison with Freeform (kUSM Q1). We ran Freeform ideas through the same executor on the nine BioXArena tasks, one run per task. On the three tasks left unchanged in the benchmark organizers’ corrected data release of 7 October 2026, Freeform scores 0.861, 0.631 and 0.771, and CAR scores 0.897, 0.624 and 0.775 (cyp-inhibition macro-AUC, breast-cancer macro-F1, pan-cancer survival c-index). The organizers rebuilt the other six tasks in that release to correct leakage, label and split problems, and we will report execution results on the corrected release in the revision.

  4. Usefulness of traces (kUSM W2/Q2, rzo3 W2). We agree. The paper limits the trace claim to procedural provenance and states that structural validity does not establish scientific correctness. Declared-entity coverage is 99.96% to 100% in the ablation arms, at least as high as CAR’s 99.70% to 100%, so it does not distinguish operator traces from ordinary logs. We will add a blinded human audit comparing operator traces with free-form logs on a sample of ideas in the next version of the paper.

AuthorsOpenReview ↗

General Response to Reviewers (2/2)

  1. Knowledge-graph quality (rzo3 W3). The paper does not measure extraction accuracy, as its limitations state. Graph construction is not claimed as a contribution. It is a deliberately simple pipeline that serves as the grounding condition. We have prepared a human audit of sampled graph facts against their source papers and will report its precision. The limit of 20 papers per topic, chosen to keep costs low, remains a stated limitation.

  2. Related work and novelty (Pi9H W1). We will add a direct comparison with SciPIP and Chain of Ideas and position the paper as a decomposition study, not as a new ideation method.

  3. Reliability of the MN metric (Pi9H W4). We agree that ρ=0.39\rho = 0.39 is modest and that the validation sample was not held out. On the LiveIdeaBench originality critic, which was not tuned on that sample, all eight main contrasts (CAR against the four baselines on both backbones) have the same sign as on MN, and no ablation contrast is significant on both backbones on either instrument. We will add blinded human ratings of a sample of proposals as a held-out check of MN.

  4. Feasibility, fluency and practical value (Pi9H W3, Q1, Q2). CAR is rated more original and less feasible than AI Scientist and AI Scientist-v2, and more feasible than Research Ideation Agent and Base, on both backbones. The paper reports this as a trade-off in which no method leads on both axes, and we will make the cost side more prominent. Fluency measures how distinct a method’s ideas are from one another, not their scientific value, and we will say this where it is reported. It is one of the five dimensions defined by LiveIdeaBench, and we report it because LLM ideation tends to repeat itself (Si et al., 2024), so a method whose ideas score well individually but are near-duplicates of each other is of limited use. The fluency advantage over Base and the two AI Scientist versions is shared by all graph-grounded conditions, including single-step Floor, so it comes from the pipeline and not from the operators. The operators do not improve the other critic metrics over free-form iteration either (no significant contrast). Execution uses three runs per task on eight FML-bench-Lite tasks and one run per task on nine BioXArena tasks. We will describe it as showing that the ideas execute competitively under a shared executor, not as evidence of practical research value.

We are grateful for the reviewers’ time and efforts. The reviews have changed how we position the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.