acceptodds
Under review as a conference paper at ICLR 2027

CoI‑Pro: Scientific Idea Generation via Retrieval over Research Trajectories

Abstract

Scientific idea generation is crucial for automated research and scientific progress. Existing methods typically follow a three-step pipeline: first constructing idea chains or graphs for a given topic, then selecting an idea generation direction, and finally producing concrete ideas. However, these methods fail to explicitly incorporate historical research trajectories from other topics, which often leads to low-quality or incremental ideas. In contrast, human researchers frequently propose innovative ideas by drawing on research trajectories from other topics, rather than confining themselves to their own topic. For instance, suppose Topic A has a research trajectory of “supervised learning → unsupervised learning → semi-supervised learning.” If a new Topic B is currently at the “unsupervised learning” stage, an automated idea generation model without referencing Topic A’s trajectory would likely remain confined to proposing ideas within unsupervised learning. Conversely, if provided with Topic A’s trajectory as a reference, the model is far more likely to generate a novel idea extending to semi-supervised learning, thereby substantially enhancing both the quality and innovativeness of the generated idea. Thus, inspired by the above intuition, we propose a novel method, called CoI-Pro, by retrieving relevant historical research trajectories in other topics to guide idea generation for a target topic. Extensive experiments show that CoI-Pro achieves average preference scores of 57.7%, 74.9%, and 58.9% against AI-Researcher, AI-Scientist-v2, and the CoI Agent, respectively, under three LLM judges on 50 temporally separated computer-science topics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scientific idea generation, automated scientific discovery, retrieval-augmented generation, research trajectory modeling, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wheCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoI-Pro, which distills citation-linked paper chains into abstract limitation–strategy trajectories and retrieves successor innovations from similar historical trajectories to guide scientific idea generation. On 50 temporally separated computer-science topics, CoI-Pro receives higher proposal preferences than three baselines under multiple LLM judges and human evaluation.

Critical strengths
  1. The motivation is intuitive, and transferring innovation patterns across topics is potentially useful.
  2. The method and overview figure are clear and easy to follow.
  3. The experiments include temporal separation, multiple judges, human validation, component ablations, and a random-retrieval control.
  4. CoI-Pro shows particularly promising improvements in feasibility and expected effectiveness.
Critical weaknesses
  1. Limited novelty. CoI-Pro largely extends existing Chain-of-Ideas and retrieval-augmented ideation methods with an additional LLM-based trajectory abstraction and retrieval stage. The conceptual advance appears incremental.

  2. The need for explicit trajectory construction is insufficiently established. Recent papers often summarize the evolution of prior work in their introductions and related-work sections. The paper does not compare against a strong baseline that receives the same literature or concrete CoI and is explicitly prompted to infer the trajectory and next direction on the fly. Thus, the improvement may primarily come from additional scaffolding and context.

  3. The role of trajectory order is not isolated. Comparisons with shuffled, order-agnostic, or last-transition-only representations are missing. Consequently, it remains unclear whether ordered trajectories are necessary or whether retrieving relevant abstract strategies is sufficient.

  4. Cost is not analyzed. CoI-Pro introduces additional GPT-4o calls for trajectory construction and distillation, but calls, tokens, latency, and monetary cost are not compared with the baselines.

  5. The evaluation is limited to judged proposals. The ideas are not executed, and the evaluation uses 50 topics, one generator backbone, apparently one generation per setting, and only three baselines. Improvements over the closest baselines are also relatively modest.

Questions for authors
  1. How does CoI-Pro compare with a model given the same concrete CoI and explicitly prompted to infer the research trajectory and next direction directly?
  2. Does performance remain when trajectory moves are shuffled, treated as unordered, or reduced to only the latest transition?
  3. What are the per-proposal API calls, token usage, latency, and monetary cost for CoI-Pro and the baselines?
  4. Are the results stable across multiple generation seeds and other generator models?
  5. Can the authors analyze whether baseline systems fail at trajectory identification, cross-topic analogy retrieval, or grounding the retrieved strategy?
15 comments
AuthorsOpenReview ↗

Response to Reviewer wheC [Part 1/6]

Thank you for your feedback and for the time and effort you put into reviewing our paper. We address your concerns below.

W1: Limited Novelty

W1. CoI-Pro largely extends existing Chain-of-Ideas and retrieval-augmented ideation methods with an additional LLM-based trajectory abstraction and retrieval stage. The conceptual advance appears incremental.

Response. We thank the reviewer for raising this concern and appreciate the opportunity to further clarify the methodological contribution of CoI-Pro.

Our contribution lies in designing Research Trajectories for retrieval and explicitly supplying the retrieved successor innovations to assist idea generation.

When developing new ideas, human researchers often draw inspiration from advances in other research fields. However, existing approaches have not explicitly implemented this process, which may lead to lower-quality research ideas. To address this, CoI-Pro first constructs a Chain of Ideas (CoI) for the target topic, then uses its Research Trajectory to retrieve similar historical prefixes and obtain their corresponding successor innovations. These successor innovations are explicitly provided to the generator and grounded in the target topic's concrete literature context to develop specific research ideas. Through this process, the model can leverage cross-topic research experience to generate higher-quality and more effective ideas.

This design differs from general retrieval-augmented ideation methods: rather than retrieving isolated related papers or generic strategies, CoI-Pro retrieves historical Research Trajectory prefixes together with their corresponding successor innovations. The retrieved prefix establishes the connection between historical experience and the target topic, while its successor provides a concrete direction for analogy.

This also differs from CoI Agent, which analyzes adjacent past steps and introduces cross-field analogy through prompting alone. CoI-Pro, in contrast, uses Research Trajectories to retrieve analogous historical experiences and explicitly supplies their associated successor innovations to the model. Rather than relying solely on the model to identify potential analogies through prompting, CoI-Pro provides retrieved historical trajectories and their subsequent innovations as explicit guidance for the analogy process.

Overall, CoI-Pro contributes at two interconnected levels: it introduces Research Trajectories, specifically prefix–successor pairs, as the retrieval unit for accessing transferable historical research experience; and it uses the retrieved successor innovations as explicit guidance for analogy-driven idea generation. This distinguishes CoI-Pro both from general retrieval-augmented ideation methods that retrieve related content and from CoI Agent, which relies on prompting alone to encourage cross-field analogy.

AuthorsOpenReview ↗

Response to Reviewer wheC [Part 2/6]

W2 & Q1: Necessity of Explicit Trajectory Construction

W2. The need for explicit trajectory construction is insufficiently established. Recent papers often summarize the evolution of prior work in their introductions and related-work sections. The paper does not compare against a strong baseline that receives the same literature or concrete CoI and is explicitly prompted to infer the trajectory and next direction on the fly. Thus, the improvement may primarily come from additional scaffolding and context.

Q1. How does CoI-Pro compare with a model given the same concrete CoI and explicitly prompted to infer the research trajectory and next direction directly?

Response. We thank the reviewer for raising this important question. The central issue is whether explicitly constructing and organizing Research Trajectories provides additional benefits over directly providing historical literature and prompting an LLM to infer research progressions and subsequent directions. To investigate this, we added an on-the-fly inference baseline.

We control the underlying historical literature available to both methods. Both CoI-Pro and the baseline receive the same research topic, the same target concrete CoI, and the paper-level information from the same five historical chains retrieved by CoI-Pro. Specifically, CoI-Pro receives its pre-distilled abstract trajectory prefixes and corresponding successor innovations. In contrast, the baseline receives the titles and abstracts of the papers in these five chains in a flat, unstructured format, without the explicitly constructed Research Trajectory structure or precomputed successor-innovation labels. This ensures that both methods have access to the same underlying papers while removing the structured trajectory guidance provided by CoI-Pro from the baseline.

For the baseline, we instruct the model to analyze the target concrete CoI and the provided paper-level information, infer the research progressions underlying the historical papers, identify potentially transferable research directions, and generate a concrete research idea accordingly. The model must therefore identify the relevant relationships and infer the next-step direction directly from the unstructured paper information.

We evaluate the two methods on 50 topics using GPT-5 as the judge, with each pair evaluated in both presentation orders. The table reports wins/ties/losses (W/T/L) and preference percentages from CoI-Pro's perspective, with ties counted as half a win.

DimensionW/T/LPreference (%)
Novelty47/2/5148.0%
Significance60/1/3960.5%
Clarity44/11/4549.5%
Feasibility61/2/3762.0%
Expected Effectiveness56/1/4356.5%
Overall Preference268/17/21555.3%

CoI-Pro achieves an Overall Preference of 55.3%, with its main advantages in significance and feasibility, as well as a positive advantage in Expected Effectiveness. The baseline performs slightly better on novelty, while the two methods achieve comparable results in clarity.

These results demonstrate the value of explicitly constructing and organizing Research Trajectories. Although both methods have access to the same underlying historical papers, CoI-Pro achieves better Overall Preference by using pre-distilled trajectory representations and their associated successor innovations. This suggests that simply providing historical papers and prompting an LLM to infer their research progressions does not fully substitute for explicitly organizing the relationships between successive research steps.

Overall, the advantage of CoI-Pro is not simply attributable to access to additional literature or paper-level information. Rather, explicitly constructing Research Trajectories provides a structured way to identify, compare, and transfer historical research progressions and their successor innovations to new idea generation.

AuthorsOpenReview ↗

Response to Reviewer wheC [Part 3/6]

W3 & Q2: The role of trajectory order

W3. The role of trajectory order is not isolated. Comparisons with shuffled, order-agnostic, or last-transition-only representations are missing. Consequently, it remains unclear whether ordered trajectories are necessary or whether retrieving relevant abstract strategies is sufficient.

Q2. Does performance remain when trajectory moves are shuffled, treated as unordered, or reduced to only the latest transition?

Response. We thank the reviewer for raising this important question. We conducted three additional ablation experiments to investigate the contribution of ordering relationships among historical innovation moves and the historical context preserved in complete Research Trajectories.

Specifically, we consider three alternative retrieval settings:

  • Shuffled: We randomly permute the order of innovation moves within each historical research chain and perform retrieval using the resulting shuffled memory, examining the effect of disrupting the original research progression.
  • Order-agnostic: We remove the exponential positional weighting scheme and assign equal weights to all innovation moves when computing trajectory relevance, examining whether the complete set of strategies remains effective without distinguishing their positions.
  • Last-transition-only: We compute retrieval relevance using only the final innovation move in each Research Trajectory, excluding all preceding moves, to examine the additional value provided by the complete historical progression.

We compare each variant against the full CoI-Pro. The table reports dimension-level results as wins/ties/losses (W/T/L) from CoI-Pro's perspective under GPT-5 judging. Overall Preference is calculated by aggregating judgments across all five dimensions, with ties counting as half a win.

CoI-Pro vs.Novelty (W/T/L)Significance (W/T/L)Clarity (W/T/L)Feasibility (W/T/L)Expected Effectiveness (W/T/L)Overall Preference (%)
Shuffled41/2/5761/4/3559/7/3455/1/4459/6/3557.0%
Order-agnostic40/2/5855/3/4249/15/3659/3/3858/0/4254.5%
Last-transition-only49/0/5152/3/4557/6/3754/0/4647/0/5352.7%

Overall, CoI-Pro achieves preference rates of 57.0%, 54.5%, and 52.7% against the Shuffled, Order-agnostic, and Last-transition-only variants, respectively.

Compared with Shuffled and Order-agnostic, CoI-Pro shows consistent advantages in feasibility and expected effectiveness. Specifically, its preference rates in feasibility are 55.5% and 60.5%, while those in expected effectiveness are 62.0% and 58.0%, respectively. However, both variants achieve higher novelty preferences than CoI-Pro (42.0% and 41.0%). This may be because disrupting the original order or removing positional distinctions allows the model to explore less constrained combinations of strategies, potentially producing ideas perceived as more novel but less grounded in coherent research progression.

Compared with Last-transition-only, the two methods perform similarly, with CoI-Pro achieving a modest overall preference of 52.7%. This suggests that the latest innovation move already captures substantial useful guidance, while retaining the complete Research Trajectory provides additional historical context and a slight overall advantage.

Reviewer_kh1dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes CoI-Pro, which extends Chain of Ideas by distilling each adjacent-paper transition into an abstract limitation–strategy "innovation move" and building a memory of historical prefix–successor records. Given a target topic, its distilled trajectory retrieves similar historical prefixes, and their observed successor innovations guide the generator's choice of research direction. On 50 post-2024 computer-science topics, CoI-Pro is preferred over AI-Researcher, AI-Scientist-v2 and CoI Agent under three LLM judges and a human evaluation.

Critical strengths
  • Clear and intuitive idea. Using "what happened next" in similar historical trajectories as guidance for direction selection is a simple, well-motivated reframing of literature retrieval for ideation, and the paper communicates it clearly.
  • Reasonably thorough evaluation. The paper uses three independent LLM judges, order-reversed pairwise comparisons, topic-level bootstrap confidence intervals, a human study with agreement analysis, a component ablation, and a random-guidance control repeated over three seeds.
Critical weaknesses
  • Limited novelty over CoI Agent [1]. The main addition is decomposing transitions into limitation–strategy pairs and retrieving prefix–successor records across topics. Similar ideas have been explored, e.g., decomposed problem/method representations with modeled ideation transitions [2] and mechanism transfer across method families [3]. CoI Agent itself already extracts limitations, analyzes adjacent transitions, and prompts for cross-field analogy. The authors should discuss these works and isolate the gain over CoI Agent's analogy prompting.

  • Undocumented data construction and an overly broad domain claim.

    • Memory: the 40 queries, their selection principle, the domain distribution, how 40 queries yield 1,541 chains, and the "validation" criteria are not reported.
    • Test set: how the 50 target papers were chosen is unstated, and the extracted topics are not human-verified.
    • Claim: the paper is framed as general scientific idea generation, but evaluation is CS-only. Claims should be scoped down or supported in other domains.
  • "Successor innovation" and cross-topic transfer are not validated. Chain construction picks the citing paper most similar to the topic, not one that addresses the predecessor's limitation, and each move is distilled independently. Successors may therefore not be genuine continuations, and the LLM may fabricate causal links. The paper also never reports how often retrieved or used records come from a different topic, so it is unclear whether gains come from cross-topic experience or just extra context.

  • Novelty does not improve, contradicting the motivation. The paper targets "incremental ideas", yet random guidance beats semantic retrieval on novelty for every seed (Table 3), and the gains concentrate on feasibility and effectiveness. The method appears to make ideas more grounded rather than more novel.

References

  • [1] Li et al. Chain of Ideas: Revolutionizing Research via Novel Idea Development with LLM Agents. Findings of EMNLP, 2025.
  • [2] Shen et al. Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas. arXiv:2601.08901, 2026.
  • [3] Hu et al. From Mechanism Discovery to Proposal Closure: Graph-Grounded Hierarchical Search for Scientific Ideation. COLM 2026.
Questions for authors
  • Why is the original target paper not included as a baseline, as in CoI Agent? It would also directly test whether retrieval steers generation toward the direction the field actually took.
  • Why is ResearchAgent, the most closely related literature-grounded method and a CoI Agent baseline, not compared?
  • The Reproducibility Statement claims the appendix gives memory construction and filtering criteria, but it only describes the temporal filter. Can the authors release the 40 queries, the chain expansion and validation procedure, the 50 target papers, and the memory?
9 comments
AuthorsOpenReview ↗

Response to Reviewer kh1d [Part 1/6]

Thank you for your feedback and for the time and effort you put into reviewing our paper. We address your concerns below.

W1: Novelty and comparison with CoI Agent

W1. CoI Agent already analyzes limitations and adjacent transitions and prompts for cross-field analogy. Related work also explores decomposed representations and mechanism transfer. What does CoI-Pro add, and how does it compare with CoI Agent’s analogy prompting?

Response. We thank the reviewer for highlighting the relationship between CoI-Pro and these existing research directions. We would like to clarify that our contribution is not simply decomposing transitions into limitation–strategy pairs, modeling ideation transitions, or prompting for cross-field analogy. Rather, our contribution lies in designing Research Trajectories for retrieval and explicitly supplying the retrieved successor innovations to assist idea generation.

When developing new ideas, human researchers often draw inspiration from advances in other research fields. However, existing approaches have not explicitly implemented this process, which may lead to lower-quality research ideas. To address this, CoI-Pro first constructs a Chain of Ideas (CoI) for the target topic, then uses its Research Trajectory to retrieve similar historical prefixes and obtain their corresponding successor innovations. These successor innovations are explicitly provided to the generator and grounded in the target topic's concrete literature context to develop specific research ideas. Through this process, the model can leverage cross-topic research experience to generate higher-quality and more effective ideas.

This differs from CoI Agent [1], which extracts limitations, analyzes adjacent transitions, and prompts for cross-field analogy, but does not retrieve historical successors through Research Trajectory matching. In [2], ideation transitions encode within-idea problem–method–findings relationships for conceptual retrieval and novelty assessment; our approach captures research progression across papers to retrieve subsequent innovations. [3] explores mechanism transfer across method families, whereas our contribution is to retrieve historical Research Trajectory prefixes together with their observed successor innovations and use them to guide idea generation.

Our comparison with CoI Agent starts from the same chains obtained in the first stage. The distinction lies in how analogy is performed afterward: CoI Agent encourages cross-field analogy through prompting, whereas CoI-Pro uses Research Trajectories to retrieve analogous historical experiences and explicitly supplies their successor innovations to the model. With the input chains held fixed, this comparison isolates the contribution of our retrieval-based, successor-guided analogy mechanism over CoI Agent's analogy prompting.

  • [1] Li et al. Chain of Ideas: Revolutionizing Research via Novel Idea Development with LLM Agents. Findings of EMNLP, 2025.
  • [2] Shen et al. Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas. arXiv:2601.08901, 2026.
  • [3] Hu et al. From Mechanism Discovery to Proposal Closure: Graph-Grounded Hierarchical Search for Scientific Ideation. COLM 2026.
AuthorsOpenReview ↗

Response to Reviewer kh1d [Part 2/6]

W2(a), W2(b) & Q3: Data Construction and Reproducibility

W2(a). The 40 queries, their selection principle, the domain distribution, how 40 queries yield 1,541 chains, and the "validation" criteria are not reported.

W2(b). How the 50 target papers were chosen is unstated, and the extracted topics are not human-verified.

Q3. The Reproducibility Statement claims the appendix gives memory construction and filtering criteria, but it only describes the temporal filter. Can the authors release the 40 queries, the chain expansion and validation procedure, the 50 target papers, and the memory?

Response.

1. Historical Memory Construction

We used 40 Semantic Scholar queries, with five queries allocated to each of eight computer science subfields, to avoid concentrating seed collection in a few research areas. Each query returned multiple candidate papers. After filtering and deduplicating the candidates, we selected seeds in round-robin order across subfields.

Starting from each seed, we followed direct references backward, retaining papers published by the end of 2024 and constructing chains of three to five papers. This process produced 1,541 concrete chains.

We retained a Research Trajectory only if every adjacent transition passed automated checks for the required nonempty fields and for overly domain-specific or implementation-specific terminology in the abstract moves. These checks ensured structural validity and screened for overly specific terminology, but did not constitute human semantic validation.

The resulting memory contains 1,512 Research Trajectories and 4,241 indexed prefix–successor records. The trajectory counts across the eight subfields are: 219 for LLMs and agents, 218 for computer vision, 205 for NLP, 196 for reinforcement learning and robotics, 176 for AI for science, 169 for security and privacy, 169 for graphs, and 160 for systems.

The https://anonymous.4open.science/r/coi-pro-review-artifacts/ provides the original query strings, subfield counts, and detailed chain-construction and filtering procedures.

2. Test-set Construction

We constructed an evaluation benchmark comprising 50 computer science research topics. Specifically, we selected target papers from publications appearing in multiple academic conferences and journals after 2024, guided by the distribution of research interests among the human evaluators. We then extracted the corresponding research topics from these papers and manually checked the extracted topics to confirm that they were appropriate and aligned with the intended research scope.

3. Artifact Release

We have released the 40 original queries, the chain-expansion and validation procedures, the subfield counts, and the complete textual retrieval memory in the https://anonymous.4open.science/r/coi-pro-review-artifacts/.

The repository includes all 1,512 Research Trajectories and 4,241 indexed records.

It also provides a list of the 50 target papers in Semantic Scholar ID format.

AuthorsOpenReview ↗

Response to Reviewer kh1d [Part 3/6]

W2(c): Disciplinary scope

W2(c). The paper is framed as general scientific idea generation, but the evaluation is restricted to computer science. The claims should be narrowed or supported by evaluations in other disciplines.

Response. We agree that the original CS-only evaluation does not establish generality across scientific disciplines. To test whether CoI-Pro extends beyond computer science, we ran additional evaluations in economics, physics, biology and medicine, and chemistry. Each field contains 20 target topics and uses a field-specific memory of 300 historical chains. The first table reports CoI-Pro’s Overall Preference against each baseline under GPT-5 judging, with ties counted as half a win.

FieldOverall Preference (%) vs. CoI AgentOverall Preference (%) vs. AI-ResearcherOverall Preference (%) vs. AI-Scientist-v2
Economics54.0%60.8%60.3%
Physics57.5%45.0%60.8%
Biology and medicine55.8%55.8%58.3%
Chemistry60.3%60.0%66.0%

The second table reports Elo ratings fitted separately for each field using all pairwise judgments among the four methods. Ratings are centered at 1,000 within each field, and the highest rating in each field is shown in bold.

FieldCoI-ProCoI AgentAI-ResearcherAI-Scientist-v2
Economics1044.41013.81009.4932.4
Physics1023.51004.11037.2935.2
Biology and medicine1034.5986.91002.6976.0
Chemistry1065.41004.71014.0916.0

CoI-Pro’s preference point estimate exceeded 50% in 11 of 12 field–baseline comparisons, and it ranked first by Elo in three of the four fields. These results provide initial evidence that CoI-Pro can be applied beyond computer science.

Reviewer_svWmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoI-Pro, a retrieval-augmented framework for scientific idea generation that uses historical research trajectories from other topics to guide the next research direction. The method constructs a citation-grounded Chain of Ideas for the target topic, distills transitions between ideas into innovation moves (a limitation and the strategy used to address it), and retrieves historical trajectory prefixes paired with their observed successor innovations. The retrieved successor guides direction selection, while the final proposal is grounded in the target topic's concrete literature chain. The memory contains 1,512 validated research trajectories and 4,241 prefix–successor records. Evaluation uses 50 computer-science topics derived from papers published after 2024 and compares CoI-Pro with AI-Researcher, AI-Scientist-v2, and the CoI Agent using three LLM judges and additional human/component analyses. The paper reports preference scores of 57.7%, 74.9%, and 58.9% against these systems in its abstract-level summary, with results generally favoring CoI-Pro.

Critical strengths
  1. The paper targets a plausible gap in research-idea generation. Retrieving not just related papers but the innovation that historically followed a similar research trajectory provides a concrete mechanism for transferring research strategies across topics.
  2. The representation of an innovation move is interpretable. Decomposing a transition into the limitation that motivated it and the strategy used to address it is more structured than retrieving whole papers or generic inspirational text.
  3. The memory construction and retrieval pipeline are explicit. The paper describes validated records and a process for grounding retrieved successors in the target topic's own Chain of Ideas. Restricting the generator to a small number of retrieved moves may help prevent proposals from becoming simple collages.
  4. The evaluation includes multiple automated-research baselines and judges. Fifty temporally separated CS topics, multiple LLM judges, component ablations, retrieval controls, and human evaluation are meaningful efforts to test more than illustrative examples.
  5. The approach has a useful practical interpretation. Rather than asking a generator to invent a direction from a large open-ended space, CoI-Pro offers a historically observed transition as a structured hypothesis for what might come next.
Critical weaknesses
  1. Novelty relative to retrieval-augmented idea generation needs sharper isolation. The central contribution is retrieving successor innovations from trajectory memory. It is not clear how much the gains depend on this successor-specific signal rather than additional retrieved context, stronger prompt structure, or the more elaborate generation pipeline. A matched retrieval baseline with the same token budget would strengthen the claim.
  2. The evaluation depends substantially on LLM-based pairwise judgments. Novelty, significance, feasibility, and expected effectiveness are difficult to assess reliably without expert review. Multiple judges are useful, but may share preferences for polished, well-structured, or familiar proposals. Agreement, uncertainty, judge-order effects, and the human evaluation protocol need detailed reporting.
  3. The benchmark is small and restricted to computer science. Fifty topics provide an initial test but cannot establish generality across scientific disciplines or research cultures. The topics are derived from recent papers; excluding target-paper content from prompts is a good precaution, but may not capture the diversity of open-ended discovery.
  4. Historical successor retrieval may encourage imitation or trajectory bias. A successor productive in one topic does not necessarily transfer to another. Retrieved moves may push generation toward familiar patterns, reducing novelty or producing plausible-sounding but technically incompatible ideas. The paper should measure whether retrieved moves are genuinely instantiated and feasible in the target domain.
  5. The memory may encode selection bias. The trajectory database is constructed from published research chains and validated records. Published literature overrepresents successful and visible directions, while failed approaches and less-established areas may be missing. This could bias the system toward popular directions rather than high-value but unconventional ones.
  6. Temporal split and leakage controls deserve scrutiny. Excluding target-paper content from the prompt is useful, but the authors should clarify whether the memory contains closely related papers, later publications, or successor ideas that effectively reveal the target's contribution. Temporal and topical separation are central to a fair evaluation.
  7. Proposal preference is not the same as scientific utility. Pairwise preference does not demonstrate that an idea can be implemented, yield reproducible results, or lead to a meaningful contribution. Claims should be calibrated to proposal quality unless downstream implementation or expert feasibility assessment is provided.
Questions for authors
  1. How much gain remains against a token-matched retrieval baseline that provides related papers or generic strategies but not explicit successor innovations?
  2. What are the inter-judge agreement statistics, confidence intervals, and details of the human evaluation, including reviewer expertise and blinding?
  3. How do you ensure that the trajectory memory contains no future or near-duplicate information that leaks the target paper's technical contribution?
  4. Can you measure whether retrieved successor innovations are faithfully instantiated in the final proposal and improve feasibility, not just preference scores?
  5. How sensitive are results to memory size, retrieval top-k, embedding model, and removal of the most popular or most similar trajectories?
  6. Does the method generalize beyond computer science, and can domain experts assess a blinded set of ideas prospectively for novelty and feasibility?
11 comments
AuthorsOpenReview ↗

Response to Reviewer svWm [Part 1/7]

Thank you for your feedback and for the time and effort you put into reviewing our paper. We address your concerns below.

W1 & Q1: Token-Matched Retrieval

W1. Novelty relative to retrieval-augmented idea generation needs sharper isolation. The central contribution is retrieving successor innovations from trajectory memory. It is not clear how much the gains depend on this successor-specific signal rather than additional retrieved context, stronger prompt structure, or the more elaborate generation pipeline. A matched retrieval baseline with the same token budget would strengthen the claim.

Q1. How much gain remains against a token-matched retrieval baseline that provides related papers or generic strategies without explicitly identifying successor innovations?

Response. We added a topic-based retrieval baseline. For each target topic, we use the five highest-ranked eligible Semantic Scholar papers published by the end of 2024 as seed papers and construct five Chains of Ideas using the procedure described in the paper, while ensuring that every paper included in each chain was published no later than 2024. These chains replace the five reference chains retrieved by CoI-Pro.

In this baseline, we use the same target topic, target CoI, and generation instructions as CoI-Pro, and present each reference chain as a whole without explicitly labeling successor innovations. In contrast, CoI-Pro explicitly identifies and supplies the retrieved successor innovations to guide research direction selection. We adjust the length of the reference texts and verify that the API-recorded input token counts differ by no more than 20 tokens for each of the 50 topics.

The table reports GPT-5 judgments from CoI-Pro's perspective as wins/ties/losses (W/T/L) and preference percentages, with ties counted as half a win.

DimensionW/T/LPreference (%)
Novelty57/4/3959.0%
Significance60/6/3463.0%
Clarity50/11/3955.5%
Feasibility49/3/4850.5%
Expected Effectiveness57/3/4058.5%
Overall Preference273/27/20057.3%

The results show that CoI-Pro's largest advantage is in significance. Overall, CoI-Pro achieves a point-estimate preference of 57.3%, with a 95% topic-bootstrap confidence interval of [49.1%, 65.2%].

By controlling for input-token counts and research-chain structure, this experiment establishes that explicitly providing successor innovations contributes to improved final generation performance.

W2, Q2 & Q6(b): Evaluation Reliability and Prospective Expert Assessment

W2. The evaluation depends substantially on LLM-based pairwise judgments. Novelty, significance, feasibility, and Expected Effectiveness are difficult to assess reliably without expert review. Multiple judges are useful, but may share preferences for polished, well-structured, or familiar proposals. Agreement, uncertainty, judge-order effects, and the human evaluation protocol need detailed reporting.

Q2. What are the inter-judge agreement statistics and confidence intervals, and what expertise and blinding procedures were used in the human evaluation?

Q6(b). Can domain experts prospectively assess a blinded set of ideas for novelty and feasibility?

Response. Our human evaluation covered 50 topics, with three CoI-Pro–baseline comparisons per topic. Each pair of proposals was evaluated in both presentation orders to balance position exposure.

We distributed the comparisons across three evaluation forms: each form contained two comparisons per topic, and each comparison appeared on two forms. Two researchers independently completed each form. This yielded four human ratings for each of the 150 distinct comparisons.

All six evaluators were computer science researchers, with expertise spanning computer vision, natural language processing, and large language models. We hid method identities and randomized the presentation order of the two proposals, labeled Idea A and Idea B. Evaluators independently judged each pair along five dimensions.

After aligning A/B labels to the underlying methods, human–human exact agreement was 71.3% (95% CI: [67.0%, 75.4%]). The dimension-level results were:

DimensionHuman–human exact agreement (%)
Novelty72.4%
Significance73.7%
Feasibility71.0%
Clarity67.0%
Expected Effectiveness72.4%
Overall71.3%
AuthorsOpenReview ↗

Response to Reviewer svWm [Part 2/7]

W3 & Q6(a): Benchmark size and disciplinary scope

W3. The benchmark is small and restricted to computer science. Fifty topics provide an initial test but cannot establish generality across scientific disciplines or research cultures. The topics are derived from recent papers; excluding target-paper content from prompts is a good precaution, but may not capture the diversity of open-ended discovery.

Response. To test whether CoI-Pro extends beyond computer science, we ran additional evaluations in economics, physics, biology and medicine, and chemistry. Each field contains 20 target topics and uses a field-specific memory of 300 historical chains. The first table reports CoI-Pro’s Overall Preference against each baseline under GPT-5 judging, with ties counted as half a win.

FieldOverall Preference (%) vs. CoI AgentOverall Preference (%) vs. AI-ResearcherOverall Preference (%) vs. AI-Scientist-v2
Economics54.0%60.8%60.3%
Physics57.5%45.0%60.8%
Biology and medicine55.8%55.8%58.3%
Chemistry60.3%60.0%66.0%

The second table reports Elo ratings fitted separately for each field using all pairwise judgments among the four methods. Ratings are centered at 1,000 within each field. The highest rating in each field is shown in bold.

FieldCoI-ProCoI AgentAI-ResearcherAI-Scientist-v2
Economics1044.41013.81009.4932.4
Physics1023.51004.11037.2935.2
Biology and medicine1034.5986.91002.6976.0
Chemistry1065.41004.71014.0916.0

CoI-Pro’s preference point estimate exceeded 50% in 11 of the 12 field–baseline comparisons, and it ranked first by Elo in three of the four fields. These results provide initial evidence that CoI-Pro can be applied beyond computer science.

We follow the standard setup used in prior work on topic-conditioned scientific idea generation: the system is given a research topic and asked to propose a concrete idea. We use each target paper only to define that topic, not as a reference answer. The generator receives the extracted topic rather than the target paper itself, and the extraction prompt instructs the model to omit the paper’s specific technical solution. This gives all methods a common starting point while leaving the choice of hypothesis and method open.

AuthorsOpenReview ↗

Response to Reviewer svWm [Part 3/7]

W4, W7 & Q4: Successor Use, Imitation, and Feasibility

W4. Historical successor retrieval may encourage imitation or trajectory bias. A successor productive in one topic does not necessarily transfer to another. Retrieved moves may push generation toward familiar patterns, reducing novelty or producing plausible-sounding but technically incompatible ideas. The paper should measure whether retrieved moves are genuinely instantiated and feasible in the target domain.

W7.Proposal preference is not the same as scientific utility. Pairwise preference does not demonstrate that an idea can be implemented, yield reproducible results, or lead to a meaningful contribution. Claims should be calibrated to proposal quality unless downstream implementation or expert feasibility assessment is provided.

Q4. Can you measure whether retrieved successor innovations are faithfully instantiated in the final proposal and improve feasibility, not just preference scores?

Response. We ran a blinded attribution study to test whether the retrieved successor innovation was reflected in the final proposal. For each of 50 topics, GPT-5 received the topic, CoI-Pro’s primary successor innovation, and five proposals in randomized order with method identities hidden. The judge selected the proposal that best instantiated the supplied innovation. The comparison includes ResearchAgent, which we added in response to Reviewer kh1d’s request.

Proposal selectedTopicsShare
CoI-Pro3774%
ResearchAgent612%
CoI Agent510%
AI-Researcher24%
AI-Scientist-v200%

The generator receives five reference chains and can choose the direction that best fits the target topic; it is not required to follow the highest-ranked successor or reproduce its original implementation. The generation prompt explicitly states (Appendix C.3.4): “Select one direction as the main guide and, only when useful, at most one additional direction. Transfer the underlying research pattern rather than copying source-domain terminology, datasets, or components.” Thus, the prompt requires the generator to choose guidance, but leaves it to the generator to adapt that guidance to the target topic and its concrete Chain of Ideas.

Regarding feasibility, human evaluation shows that CoI-Pro achieves an average preference of 67.3% over the other baselines, with ties counted as half a win.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers: Summary of Revisions in the Updated Manuscript

We sincerely thank the reviewers for their constructive and insightful feedback. In response to the comments, we have substantially expanded the supplementary material to provide more comprehensive evaluations and analyses of CoI-Pro. These additions aim to further establish the reliability, generalizability, robustness, and efficiency of our approach, as well as to provide a more thorough understanding of the contributions of research trajectory construction and retrieval.

Specifically, we have made the following additions and clarifications:

  1. Human evaluation and evaluation reliability (Appendix G). We have provided a more detailed description of the human evaluation protocol, including the evaluation procedure, presentation order, and annotator assignment. We have also reported human–human agreement to further assess the reliability of the evaluation.

  2. Cross-field generalizability (Appendix H). We have extended our evaluation to four additional scientific fields: economics, physics, biology and medicine, and chemistry, with 50 topics in each field, to examine the applicability of CoI-Pro beyond computer science.

  3. Retrieval sensitivity analysis (Appendix I). We have conducted additional experiments to examine the effects of memory size, retrieval depth, and embedding model choices on the generated proposals.

  4. Additional baseline comparisons (Appendix J). We have included the original ideas presented in published target papers as reference candidates and added ResearchAgent as an additional baseline. We have also updated the Elo ranking based on the expanded set of pairwise comparisons.

  5. Computational cost analysis (Appendix K). We have reported the end-to-end computational costs of the compared systems, including monetary costs, LLM and literature API calls, token consumption, and execution time.

  6. Robustness across generation seeds and models (Appendix L). We have conducted additional experiments across eight generation seeds and two alternative generator models, Qwen-Max and DeepSeek-V3, to examine the stability of CoI-Pro's performance.

  7. Further ablation studies on research trajectories (Appendix M). We have expanded the ablation analysis to separately investigate the effects of trajectory order during retrieval, historical context during generation, and explicit limitation–strategy distillation. We have also added a controlled comparison between explicit trajectory construction and on-the-fly inference, clarifying the scope of this comparison.

  8. Token-matched retrieval comparison (Appendix N). We have added a comparison against topic-based chain retrieval under a matched input-token budget, providing further evidence for the benefit of explicitly retrieving and providing successor innovations.

These additions provide a more comprehensive evaluation of CoI-Pro from multiple perspectives and address the reviewers' concerns regarding evaluation reliability, generalizability, retrieval effectiveness, baseline comparisons, computational costs, and the individual contributions of our design choices.

For ease of review, we have highlighted all newly added content in blue in the revised manuscript, while retaining the original content in black.

We sincerely appreciate the reviewers' valuable suggestions, which have helped us improve the completeness and clarity of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.