- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CoI-Pro, which distills citation-linked paper chains into abstract limitation–strategy trajectories and retrieves successor innovations from similar historical trajectories to guide scientific idea generation. On 50 temporally separated computer-science topics, CoI-Pro receives higher proposal preferences than three baselines under multiple LLM judges and human evaluation.
Critical strengths
- The motivation is intuitive, and transferring innovation patterns across topics is potentially useful.
- The method and overview figure are clear and easy to follow.
- The experiments include temporal separation, multiple judges, human validation, component ablations, and a random-retrieval control.
- CoI-Pro shows particularly promising improvements in feasibility and expected effectiveness.
Critical weaknesses
-
Limited novelty. CoI-Pro largely extends existing Chain-of-Ideas and retrieval-augmented ideation methods with an additional LLM-based trajectory abstraction and retrieval stage. The conceptual advance appears incremental.
-
The need for explicit trajectory construction is insufficiently established. Recent papers often summarize the evolution of prior work in their introductions and related-work sections. The paper does not compare against a strong baseline that receives the same literature or concrete CoI and is explicitly prompted to infer the trajectory and next direction on the fly. Thus, the improvement may primarily come from additional scaffolding and context.
-
The role of trajectory order is not isolated. Comparisons with shuffled, order-agnostic, or last-transition-only representations are missing. Consequently, it remains unclear whether ordered trajectories are necessary or whether retrieving relevant abstract strategies is sufficient.
-
Cost is not analyzed. CoI-Pro introduces additional GPT-4o calls for trajectory construction and distillation, but calls, tokens, latency, and monetary cost are not compared with the baselines.
-
The evaluation is limited to judged proposals. The ideas are not executed, and the evaluation uses 50 topics, one generator backbone, apparently one generation per setting, and only three baselines. Improvements over the closest baselines are also relatively modest.
Questions for authors
- How does CoI-Pro compare with a model given the same concrete CoI and explicitly prompted to infer the research trajectory and next direction directly?
- Does performance remain when trajectory moves are shuffled, treated as unordered, or reduced to only the latest transition?
- What are the per-proposal API calls, token usage, latency, and monetary cost for CoI-Pro and the baselines?
- Are the results stable across multiple generation seeds and other generator models?
- Can the authors analyze whether baseline systems fail at trajectory identification, cross-topic analogy retrieval, or grounding the retrieved strategy?
15 comments
Response to Reviewer wheC [Part 2/6]
W2 & Q1: Necessity of Explicit Trajectory Construction
W2. The need for explicit trajectory construction is insufficiently established. Recent papers often summarize the evolution of prior work in their introductions and related-work sections. The paper does not compare against a strong baseline that receives the same literature or concrete CoI and is explicitly prompted to infer the trajectory and next direction on the fly. Thus, the improvement may primarily come from additional scaffolding and context.
Q1. How does CoI-Pro compare with a model given the same concrete CoI and explicitly prompted to infer the research trajectory and next direction directly?
Response. We thank the reviewer for raising this important question. The central issue is whether explicitly constructing and organizing Research Trajectories provides additional benefits over directly providing historical literature and prompting an LLM to infer research progressions and subsequent directions. To investigate this, we added an on-the-fly inference baseline.
We control the underlying historical literature available to both methods. Both CoI-Pro and the baseline receive the same research topic, the same target concrete CoI, and the paper-level information from the same five historical chains retrieved by CoI-Pro. Specifically, CoI-Pro receives its pre-distilled abstract trajectory prefixes and corresponding successor innovations. In contrast, the baseline receives the titles and abstracts of the papers in these five chains in a flat, unstructured format, without the explicitly constructed Research Trajectory structure or precomputed successor-innovation labels. This ensures that both methods have access to the same underlying papers while removing the structured trajectory guidance provided by CoI-Pro from the baseline.
For the baseline, we instruct the model to analyze the target concrete CoI and the provided paper-level information, infer the research progressions underlying the historical papers, identify potentially transferable research directions, and generate a concrete research idea accordingly. The model must therefore identify the relevant relationships and infer the next-step direction directly from the unstructured paper information.
We evaluate the two methods on 50 topics using GPT-5 as the judge, with each pair evaluated in both presentation orders. The table reports wins/ties/losses (W/T/L) and preference percentages from CoI-Pro's perspective, with ties counted as half a win.
| Dimension | W/T/L | Preference (%) |
|---|---|---|
| Novelty | 47/2/51 | 48.0% |
| Significance | 60/1/39 | 60.5% |
| Clarity | 44/11/45 | 49.5% |
| Feasibility | 61/2/37 | 62.0% |
| Expected Effectiveness | 56/1/43 | 56.5% |
| Overall Preference | 268/17/215 | 55.3% |
CoI-Pro achieves an Overall Preference of 55.3%, with its main advantages in significance and feasibility, as well as a positive advantage in Expected Effectiveness. The baseline performs slightly better on novelty, while the two methods achieve comparable results in clarity.
These results demonstrate the value of explicitly constructing and organizing Research Trajectories. Although both methods have access to the same underlying historical papers, CoI-Pro achieves better Overall Preference by using pre-distilled trajectory representations and their associated successor innovations. This suggests that simply providing historical papers and prompting an LLM to infer their research progressions does not fully substitute for explicitly organizing the relationships between successive research steps.
Overall, the advantage of CoI-Pro is not simply attributable to access to additional literature or paper-level information. Rather, explicitly constructing Research Trajectories provides a structured way to identify, compare, and transfer historical research progressions and their successor innovations to new idea generation.
Response to Reviewer wheC [Part 3/6]
W3 & Q2: The role of trajectory order
W3. The role of trajectory order is not isolated. Comparisons with shuffled, order-agnostic, or last-transition-only representations are missing. Consequently, it remains unclear whether ordered trajectories are necessary or whether retrieving relevant abstract strategies is sufficient.
Q2. Does performance remain when trajectory moves are shuffled, treated as unordered, or reduced to only the latest transition?
Response. We thank the reviewer for raising this important question. We conducted three additional ablation experiments to investigate the contribution of ordering relationships among historical innovation moves and the historical context preserved in complete Research Trajectories.
Specifically, we consider three alternative retrieval settings:
- Shuffled: We randomly permute the order of innovation moves within each historical research chain and perform retrieval using the resulting shuffled memory, examining the effect of disrupting the original research progression.
- Order-agnostic: We remove the exponential positional weighting scheme and assign equal weights to all innovation moves when computing trajectory relevance, examining whether the complete set of strategies remains effective without distinguishing their positions.
- Last-transition-only: We compute retrieval relevance using only the final innovation move in each Research Trajectory, excluding all preceding moves, to examine the additional value provided by the complete historical progression.
We compare each variant against the full CoI-Pro. The table reports dimension-level results as wins/ties/losses (W/T/L) from CoI-Pro's perspective under GPT-5 judging. Overall Preference is calculated by aggregating judgments across all five dimensions, with ties counting as half a win.
| CoI-Pro vs. | Novelty (W/T/L) | Significance (W/T/L) | Clarity (W/T/L) | Feasibility (W/T/L) | Expected Effectiveness (W/T/L) | Overall Preference (%) |
|---|---|---|---|---|---|---|
| Shuffled | 41/2/57 | 61/4/35 | 59/7/34 | 55/1/44 | 59/6/35 | 57.0% |
| Order-agnostic | 40/2/58 | 55/3/42 | 49/15/36 | 59/3/38 | 58/0/42 | 54.5% |
| Last-transition-only | 49/0/51 | 52/3/45 | 57/6/37 | 54/0/46 | 47/0/53 | 52.7% |
Overall, CoI-Pro achieves preference rates of 57.0%, 54.5%, and 52.7% against the Shuffled, Order-agnostic, and Last-transition-only variants, respectively.
Compared with Shuffled and Order-agnostic, CoI-Pro shows consistent advantages in feasibility and expected effectiveness. Specifically, its preference rates in feasibility are 55.5% and 60.5%, while those in expected effectiveness are 62.0% and 58.0%, respectively. However, both variants achieve higher novelty preferences than CoI-Pro (42.0% and 41.0%). This may be because disrupting the original order or removing positional distinctions allows the model to explore less constrained combinations of strategies, potentially producing ideas perceived as more novel but less grounded in coherent research progression.
Compared with Last-transition-only, the two methods perform similarly, with CoI-Pro achieving a modest overall preference of 52.7%. This suggests that the latest innovation move already captures substantial useful guidance, while retaining the complete Research Trajectory provides additional historical context and a slight overall advantage.
Response to Reviewer wheC [Part 1/6]
Thank you for your feedback and for the time and effort you put into reviewing our paper. We address your concerns below.
W1: Limited Novelty
W1. CoI-Pro largely extends existing Chain-of-Ideas and retrieval-augmented ideation methods with an additional LLM-based trajectory abstraction and retrieval stage. The conceptual advance appears incremental.
Response. We thank the reviewer for raising this concern and appreciate the opportunity to further clarify the methodological contribution of CoI-Pro.
Our contribution lies in designing Research Trajectories for retrieval and explicitly supplying the retrieved successor innovations to assist idea generation.
When developing new ideas, human researchers often draw inspiration from advances in other research fields. However, existing approaches have not explicitly implemented this process, which may lead to lower-quality research ideas. To address this, CoI-Pro first constructs a Chain of Ideas (CoI) for the target topic, then uses its Research Trajectory to retrieve similar historical prefixes and obtain their corresponding successor innovations. These successor innovations are explicitly provided to the generator and grounded in the target topic's concrete literature context to develop specific research ideas. Through this process, the model can leverage cross-topic research experience to generate higher-quality and more effective ideas.
This design differs from general retrieval-augmented ideation methods: rather than retrieving isolated related papers or generic strategies, CoI-Pro retrieves historical Research Trajectory prefixes together with their corresponding successor innovations. The retrieved prefix establishes the connection between historical experience and the target topic, while its successor provides a concrete direction for analogy.
This also differs from CoI Agent, which analyzes adjacent past steps and introduces cross-field analogy through prompting alone. CoI-Pro, in contrast, uses Research Trajectories to retrieve analogous historical experiences and explicitly supplies their associated successor innovations to the model. Rather than relying solely on the model to identify potential analogies through prompting, CoI-Pro provides retrieved historical trajectories and their subsequent innovations as explicit guidance for the analogy process.
Overall, CoI-Pro contributes at two interconnected levels: it introduces Research Trajectories, specifically prefix–successor pairs, as the retrieval unit for accessing transferable historical research experience; and it uses the retrieved successor innovations as explicit guidance for analogy-driven idea generation. This distinguishes CoI-Pro both from general retrieval-augmented ideation methods that retrieve related content and from CoI Agent, which relies on prompting alone to encourage cross-field analogy.