- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how the organization of external scientific knowledge influences LLM-based scientific ideation by explicitly decomposing knowledge organization into knowledge access and knowledge presentation. It first studies single-step hypothesis generation on ResearchBench through controlled comparisons between hybrid retrieval and graph-hop retrieval, as well as between linear and relation-grouped presentations. It then extends the analysis to a four-step refinement and recombination process on CHIMERA, covering three generator models and two judge models. The results show that differences in scientific ideation are driven primarily by the evidence that actually reaches the model, while presentation format has limited impact when the evidence is held fixed. In multi-step development, recombination instructions substantially increase the use of cross-branch knowledge.
Critical strengths
-
The paper decomposes knowledge organization into knowledge access and knowledge presentation, which avoids the multi-factor confounding that commonly arises when directly comparing complete RAG or graph-based systems.
-
The experimental design is rigorous, with the main findings evaluated across multiple generator and judge models, as well as different candidate pool sizes and token budgets.
Critical weaknesses
-
The paper uses only relatively small subsets of ResearchBench and CHIMERA and compare only two knowledge access methods and two knowledge presentation formats. The lack of controlled comparisons with broader knowledge organization paradigms, such as HippoRAG, RAPTOR, or hierarchical LLM-Wiki-style approaches, limits the generalizability of the main conclusions.
-
Another major concern is that the static study primarily evaluates reference-hypothesis matching, while the dynamic study focuses on cross-branch citation. Neither directly measures idea novelty, feasibility, or scientific value, so the conclusions may overgeneralize to scientific ideation quality more broadly.
Questions for authors
-
Could the authors add human evaluation or independent quality metrics to verify whether the main findings also hold for research idea quality?
-
Since the current pool expansion adds only non-inspiration papers and graph-hop exhibits substantial budget under-utilization in some settings, could the authors evaluate more natural pool-expansion strategies and budget-matched retrieval settings to strengthen the conclusions?
-
Some generator–judge pairings involve models from the same family, which may introduce model-family bias. Could the authors expand blind human validation across outputs from different generators and report the number and background of annotators, annotation time, and cost to more thoroughly assess judge reliability?
3 comments
Thanks for your rebuttal. While I still find the experimental evidence remain limited, particularly due to the lack of direct comparisons with other knowledge organization paradigms as baselines, which weakens the generalizability of the conclusions, I am willing to maintain my rating. I wish the authors the best of luck!
Thank you for your follow-up and for maintaining your positive rating. We appreciate you highlighting this direction: direct comparisons with complete knowledge organization systems such as HippoRAG and RAPTOR would be a valuable complement to our controlled analysis. Our study was designed to hold the generator, candidate pool, budget, and prompt fixed, so that it isolates the effects of access and presentation, which complete systems change together. Within this design, the conclusions hold across three generators, two judges, multiple pool sizes and budgets, and the additional experiments reported in our rebuttal. In the revision, we will state this scope explicitly in the limitations section and explain how paradigms such as HippoRAG and RAPTOR map onto our access–presentation framework, so that future work can apply the same controls to them. Thank you again for your constructive suggestions and kind wishes.
We truly appreciate your time, effort, and valuable guidance. Your suggestions were very concrete, and we have completed all of the suggested experiments.We respond point by point below.
W1: Sample coverage and broader organization paradigms
Thank you for the reminder. In terms of scale, the 240 tasks of the static study cover all 12 disciplines in ResearchBench, the dynamic study uses 130 primary CHIMERA tasks stratified by relation type and by whether they cross domains, and the main experiments generated more than 68,000 ideas in total. In terms of breadth, we conducted three additional sets of experiments: random candidate pools and budget matching broaden the comparison of retrieval methods, cross-paper graph layouts broaden the comparison of presentation. All conclusions remain consistent. Systems such as HippoRAG and RAPTOR mainly change which evidence is placed in the context, which is exactly the side our paper finds to matter, so our analysis can be applied directly to explain the sources of their gains. We also plan to apply the same controls to HippoRAG, RAPTOR, and similar systems in future work.
W2 & Q1: Human evaluation of idea quality
Thank you for the suggestion. We added a blind human quality comparison of 100 proposal pairs by two raters (with large disagreements re-checked by a third rater), covering novelty, soundness, feasibility, and scientific value. The results point in the same direction as the original metrics: changing the presentation does not affect idea quality, hybrid retrieval is better on three dimensions, and recombination is better on all four, with the feasibility difference significant at the 95% level.
Table R4a. Blind quality comparison and substantive content differences for 100 proposal pairs
Q2: More natural pool expansion and budget-matched retrieval
Both suggestions were very helpful, and we have completed the corresponding experiments (Table R4b). Compared with randomly sampled 25-paper candidate sets, using the full 75-paper candidate set did not lower scores, showing that natural pool expansion does not dilute the evidence. Meanwhile, at the same size of 25 papers, the pool that keeps all inspiration papers scored 0.22–0.37 higher than the random ones. On the budget side, graph-hop slightly outperformed hybrid retrieval once it used its full budget, and the gap between the two disappeared once the amount of evidence placed in the context was matched. These results show more clearly that what matters is the relevant evidence placed in the context, consistent with our conclusions. We will revise the wording in the paper accordingly, stating that the effect of pool expansion depends on the relevant evidence placed in the context.
Table R4b. Natural pool expansion and budget matching
Q3: Model-family bias and human validation
Thank you for your attention to the reliability of our evaluation. We added two human raters and a third-rater check to examine scoring consistency and family preferences. Of the four generator–judge pairings, only one is within the same model family, and the other three cross-family pairings yield the same conclusions. Replacing the LLM judge with DeepSeek-Flash on the same outputs changes the presentation comparison by only 0.04. Our new human validation covers 72 outputs from all three generators; the two raters agree at an ICC of 0.83, and the same-family pairings show no consistent bias. The number, background, and time of the annotators are given in Table R4c.
Table R4c. Annotators and annotation cost
Thank you again for your detailed and constructive suggestions.