acceptodds
Under review as a conference paper at ICLR 2027

How Does Knowledge Organization Shape LLM-Based Scientific Ideation

Abstract

AI scientists powered by large language models (LLMs) draw on domain knowledge to propose and develop research ideas. Existing systems organize this knowledge differently, as retrieved papers, literature chains, or concept graphs. Comparing complete systems cannot reveal which organizational choice shapes the resulting ideas, because such comparisons change the delivered evidence and its arrangement at once. We introduce a controlled evaluation framework that separates these two factors: knowledge access (what evidence reaches the model under a fixed budget) and knowledge presentation (how the same evidence is arranged). Across two studies with three generators and two judges, we run more than 68,000 idea generations and 87,000 model evaluations. On ResearchBench, access methods change the evidence the model receives and, where that evidence differs most, the benchmark-matching score. Enlarging the candidate pool dilutes the relevant evidence admitted and does not improve matching. With the evidence held fixed, linear and relation-grouped presentations are practically equivalent in all four generator–judge pairings. In multi-step development built on CHIMERA, a recombination instruction raises the rate of explicit citation of knowledge from other research branches by 52–62 percentage points over refinement, whereas two access methods delivering similar evidence show no consistent difference. These results indicate that knowledge organization shapes the resulting ideas mainly through which evidence reaches the model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scientific ideation, large language models, knowledge organization, AI scientists

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_W82JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the organization of external scientific knowledge influences LLM-based scientific ideation by explicitly decomposing knowledge organization into knowledge access and knowledge presentation. It first studies single-step hypothesis generation on ResearchBench through controlled comparisons between hybrid retrieval and graph-hop retrieval, as well as between linear and relation-grouped presentations. It then extends the analysis to a four-step refinement and recombination process on CHIMERA, covering three generator models and two judge models. The results show that differences in scientific ideation are driven primarily by the evidence that actually reaches the model, while presentation format has limited impact when the evidence is held fixed. In multi-step development, recombination instructions substantially increase the use of cross-branch knowledge.

Critical strengths
  1. The paper decomposes knowledge organization into knowledge access and knowledge presentation, which avoids the multi-factor confounding that commonly arises when directly comparing complete RAG or graph-based systems.

  2. The experimental design is rigorous, with the main findings evaluated across multiple generator and judge models, as well as different candidate pool sizes and token budgets.

Critical weaknesses
  1. The paper uses only relatively small subsets of ResearchBench and CHIMERA and compare only two knowledge access methods and two knowledge presentation formats. The lack of controlled comparisons with broader knowledge organization paradigms, such as HippoRAG, RAPTOR, or hierarchical LLM-Wiki-style approaches, limits the generalizability of the main conclusions.

  2. Another major concern is that the static study primarily evaluates reference-hypothesis matching, while the dynamic study focuses on cross-branch citation. Neither directly measures idea novelty, feasibility, or scientific value, so the conclusions may overgeneralize to scientific ideation quality more broadly.

Questions for authors
  1. Could the authors add human evaluation or independent quality metrics to verify whether the main findings also hold for research idea quality?

  2. Since the current pool expansion adds only non-inspiration papers and graph-hop exhibits substantial budget under-utilization in some settings, could the authors evaluate more natural pool-expansion strategies and budget-matched retrieval settings to strengthen the conclusions?

  3. Some generator–judge pairings involve models from the same family, which may introduce model-family bias. Could the authors expand blind human validation across outputs from different generators and report the number and background of annotators, annotation time, and cost to more thoroughly assess judge reliability?

3 comments

We truly appreciate your time, effort, and valuable guidance. Your suggestions were very concrete, and we have completed all of the suggested experiments.We respond point by point below.

W1: Sample coverage and broader organization paradigms

Thank you for the reminder. In terms of scale, the 240 tasks of the static study cover all 12 disciplines in ResearchBench, the dynamic study uses 130 primary CHIMERA tasks stratified by relation type and by whether they cross domains, and the main experiments generated more than 68,000 ideas in total. In terms of breadth, we conducted three additional sets of experiments: random candidate pools and budget matching broaden the comparison of retrieval methods, cross-paper graph layouts broaden the comparison of presentation. All conclusions remain consistent. Systems such as HippoRAG and RAPTOR mainly change which evidence is placed in the context, which is exactly the side our paper finds to matter, so our analysis can be applied directly to explain the sources of their gains. We also plan to apply the same controls to HippoRAG, RAPTOR, and similar systems in future work.

W2 & Q1: Human evaluation of idea quality

Thank you for the suggestion. We added a blind human quality comparison of 100 proposal pairs by two raters (with large disagreements re-checked by a third rater), covering novelty, soundness, feasibility, and scientific value. The results point in the same direction as the original metrics: changing the presentation does not affect idea quality, hybrid retrieval is better on three dimensions, and recombination is better on all four, with the feasibility difference significant at the 95% level.

Table R4a. Blind quality comparison and substantive content differences for 100 proposal pairs

ItemGrouped vs. linear (40 pairs)Hybrid vs. graph-hop (30 pairs)Recombination vs. refinement (30 pairs)
Substantive difference (both raters)Only details change25/3028/30
Novelty preference+0.01+0.17+0.22
Soundness preference+0.13+0.12+0.30
Feasibility preference−0.01−0.03+0.27
Scientific value preference+0.11+0.23+0.23

Q2: More natural pool expansion and budget-matched retrieval

Both suggestions were very helpful, and we have completed the corresponding experiments (Table R4b). Compared with randomly sampled 25-paper candidate sets, using the full 75-paper candidate set did not lower scores, showing that natural pool expansion does not dilute the evidence. Meanwhile, at the same size of 25 papers, the pool that keeps all inspiration papers scored 0.22–0.37 higher than the random ones. On the budget side, graph-hop slightly outperformed hybrid retrieval once it used its full budget, and the gap between the two disappeared once the amount of evidence placed in the context was matched. These results show more clearly that what matters is the relevant evidence placed in the context, consistent with our conclusions. We will revise the wording in the paper accordingly, stating that the effect of pool expansion depends on the relevant evidence placed in the context.

Table R4b. Natural pool expansion and budget matching

ComparisonScore diff. (min)Score diff. (max)Note
Full 75-paper pool − random 25+0.13+0.14Not significant
Original 25-paper pool − random 25+0.22+0.37Significant
Graph-hop + dense backfill − hybrid+0.10+0.15Not significant
Truncated hybrid − graph-hop−0.03+0.16Not significant

Q3: Model-family bias and human validation

Thank you for your attention to the reliability of our evaluation. We added two human raters and a third-rater check to examine scoring consistency and family preferences. Of the four generator–judge pairings, only one is within the same model family, and the other three cross-family pairings yield the same conclusions. Replacing the LLM judge with DeepSeek-Flash on the same outputs changes the presentation comparison by only 0.04. Our new human validation covers 72 outputs from all three generators; the two raters agree at an ICC of 0.83, and the same-family pairings show no consistent bias. The number, background, and time of the annotators are given in Table R4c.

Table R4c. Annotators and annotation cost

StudyAnnotatorsSampleHours per rater
New: blind quality comparison2 raters (interdisciplinary CS background) + third-rater check100 proposal pairs36 / 31
New: judge validation2 raters (interdisciplinary CS background) + third-rater check72 outputs10 / 11
New: mechanism-use check2 raters (interdisciplinary CS background) + third-rater check96 proposals38 / 39
New: partner-content checkSame as above80 proposals16 / 15
New: cross-paper relation checkSame as above100 relations17 / 15

Thank you again for your detailed and constructive suggestions.

Thanks for your rebuttal. While I still find the experimental evidence remain limited, particularly due to the lack of direct comparisons with other knowledge organization paradigms as baselines, which weakens the generalizability of the conclusions, I am willing to maintain my rating. I wish the authors the best of luck!

Thank you for your follow-up and for maintaining your positive rating. We appreciate you highlighting this direction: direct comparisons with complete knowledge organization systems such as HippoRAG and RAPTOR would be a valuable complement to our controlled analysis. Our study was designed to hold the generator, candidate pool, budget, and prompt fixed, so that it isolates the effects of access and presentation, which complete systems change together. Within this design, the conclusions hold across three generators, two judges, multiple pool sizes and budgets, and the additional experiments reported in our rebuttal. In the revision, we will state this scope explicitly in the limitations section and explain how paradigms such as HippoRAG and RAPTOR map onto our access–presentation framework, so that future work can apply the same controls to them. Thank you again for your constructive suggestions and kind wishes.

Reviewer_7QT8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The main focus of this paper is the impact of knowledge organization on scientific idea generation by large language models. To analyze the effects of different knowledge organization methods on generation outcomes, the authors propose a controlled experimental framework that distinguishes between Knowledge Access and Knowledge Presentation. The paper conducted a variety of experiments, and the results indicate that knowledge organization primarily influences the generation of scientific ideas by affecting the evidence available to the model. The method of access affects the quantity and relevance of the evidence, while the form of presentation has little effect; however, knowledge reorganization instructions can significantly increase the proportion of cross-domain citations.

Critical strengths

1.Unlike previous studies, this paper identifies knowledge acquisition and knowledge presentation as two independent variables and controls for related factors to more clearly demonstrate the impact of external knowledge conditions on the results, thereby holding practical significance and methodological value.

2.The paper employs multiple generative and evaluation models, involves both dynamic and static experiments, and utilizes paired comparison confidence intervals and corrections for multiple comparisons, thereby enhancing the robustness of the experiments and the interpretability of the results.

Critical weaknesses

1.In the article, the effectiveness of static experiments is measured by the degree of alignment between the generated hypothesis and the reference hypothesis, while dynamic experiments use explicit citations across disciplines as the primary metric. There is a significant gap between the core evaluation metrics used and the quality of scientific creativity, and these metrics do not fully support the article’s central argument.

2.It is difficult to determine whether the performance changes described in the article are attributable to the knowledge organization structure itself; the results may be influenced by factors such as the number of search results, evidence coverage, and the model’s ability to follow instructions.

Questions for authors

1.Since the metrics presented in the paper do not directly reflect the soundness or novelty of scientific ideas, could the authors provide additional evaluations by human experts or other more representative metrics?

2.The experimental instructions require citing literature from other research fields, but the primary evaluation metric is also the proportion of cross-disciplinary citations. Does this mean that the cited cross-disciplinary knowledge actually contributes to the construction of scientific mechanisms? Is it necessary to further distinguish between compliance with the instructions and substantive knowledge integration?

1 comment

We sincerely thank you for your careful review and guidance, for the time and effort you have invested in our paper, and for your valuable comments. Your concern about the validity of our metrics is well taken, and we have added two human studies to address it. We respond point by point below.

W1 & Q1: Evaluating research quality; can human evaluation be added?

We added a human evaluation of 100 proposal pairs by two raters (with large disagreements re-checked by a third rater), covering three types of comparison: presentation, access, and the recombination instruction. The raters judged novelty, soundness, feasibility, and scientific value, and also marked whether the content changed substantively (Table R3a). The results point in the same direction as the original metrics. Changing the presentation does not affect idea quality, hybrid retrieval outperforms graph-hop on novelty, soundness, and scientific value, and recombination outperforms refinement on all four dimensions, with the feasibility difference significant at the 95% level.

Table R3a. Blind quality comparison and substantive content differences for 100 proposal pairs

ItemGrouped vs. linear (40 pairs)Hybrid vs. graph-hop (30 pairs)Recombination vs. refinement (30 pairs)
Substantive difference (both raters)Only details change25/3028/30
Novelty preference+0.01+0.17+0.22
Soundness preference+0.13+0.12+0.30
Feasibility preference−0.01−0.03+0.27
Scientific value preference+0.11+0.23+0.23

W2: Results may stem from retrieval quantity, evidence coverage, or instruction following

Thank you for pointing this out. This is in fact the very mechanism our paper aims to reveal: knowledge organization acts mainly through how much evidence reaches the model's context and whether it is relevant. We ran additional experiments to test these three factors one by one. When comparing retrieval methods, the gap disappears as soon as both deliver the same amount of evidence to the context. With the amount of evidence held fixed, replacing it with irrelevant content lowers scores by 0.19–0.43, showing that relevance itself also matters. As for instruction following, the instructions are identical within every comparison, and no more than 2 of each generator's 7,680 outputs contained formatting errors.

Q2: Is cross-domain citation instruction compliance or substantive knowledge integration?

Thank you for this excellent question; we agree that the two need to be distinguished. To this end, we drew a stratified sample of 96 from 3,120 experimental outputs and asked two human raters to check, against all literature from other branches that the model had seen (with large disagreements re-checked by a third rater), whether the method in each proposal actually uses a mechanism from that literature. A mechanism counts as used only if removing it would leave the method missing a key step (Table R3b). Both raters judged that all 72 proposals with cross-domain citations substantively use the cited mechanisms, and that none of the 24 proposals without such citations do, even though 22 of them had in fact seen this literature. This shows that the metric measures whether the model actually uses external knowledge, not merely whether it follows the instruction.

Table R3b. Does cross-branch knowledge actually enter proposal construction?

CheckRater ARater B
72 citing proposals: use ≥1 shown cross-branch mechanism72/7272/72
Same 72 proposals: use ≥1 cited cross-branch mechanism72/7270/72
24 non-citing proposals: still use a cross-branch mechanism0/240/24
Extrapolated to all citing proposals: share using ≥1 cited mechanism100% (95% CI 64–100%)97% (95% CI 60–99.9%)

Thank you again for the time and effort you have invested, and for your valuable and constructive comments. The human studies make our conclusions more credible, and we will add them to the revision. We hope the responses above resolve your concerns about the validity of the metrics.

Reviewer_USGKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper examines an important question on how the organization of retrieved external evidence, particularly scientific papers, affect LLM-based scientific idea generation. The authors observe that existing retrieval-augmented idea generation systems often claim that structured knowledge improves ideation, yet these systems cannot be compared directly because each one differs both in how it retrieves evidence and in how it presents that evidence to the model. To disentangle these effects, the authors decompose evidence organization into two factors: (i) presentation, the layout and structure of the retrieved evidence, and (ii) access, the method used to retrieve that evidence from a large pool of papers. Holding the idea generator fixed, they evaluate two settings drawn from existing benchmarks: direct ideation from retrieved papers and multi-step ideation. The study spans three generator models and two judge models, totaling 68,000 generated ideas. The results show that presentation has little effect on the generated ideas; instead, access, in terms of both the relevance and the amount of retrieved evidence, is what primarily drives the ideation performance of current language models.

Critical strengths

1) Well-Motivated Question: The authors address an important yet underexplored question in automated scientific ideation. Although prior work has shown that structured external evidence can guide automated ideation, it remains unclear whether the resulting performance gains come from how the evidence is structured or from the content of the evidence itself. Disentangling these two factors is important for guiding the design of future automated scientific ideation systems.

2) Systematic Analysis Across Diverse Evaluation Settings: The paper clarifies the previously vague notion of “organization of scientific evidence” by decomposing it into two factors: (i) access and (ii) presentation. It then examines the effect of each factor through well-controlled experiments in two ideation settings adapted from existing benchmarks, using three generator models and two judge models. Together, these form a robust evaluation suite.

Critical weaknesses

1) Limited Realism of the Experimental Setting The knowledge structures and graph-based access methods examined in this paper are considerably simpler than those in the prior work it cites. For example, Chain-of-Ideas links papers into chains of ideas that span multiple papers, and GraphRAG and HippoRAG similarly build relations across multiple papers. In contrast, the relations used in this study are confined to a single paper. By design, this makes regrouping them likely to have little or no effect (Figure 3c), and it also explains the marginal differences in the relation-removal control study (Table 7). It is therefore unclear whether the findings on evidence presentation would hold for relations that connect multiple papers within more complex, realistic knowledge structures. In addition, Table 1 compares only two access methods and does not include the graph structures used in prior work. As a result, it is unclear whether the findings generalize to realistic ideation systems that use graph-structured evidence.

2) Insufficient Discussion of the Generator’s Internal Knowledge, Compounded by Limited Metrics The paper does not analyze how the generator’s internal knowledge affects the results. It also gives too little detail on the knowledge cutoffs of the generator models and the publication years of the papers used for evaluation, especially for ResearchBench. If a model already knows the reference hypothesis, the evidence may act as a cue for recall rather than as a real source of inspiration. Heavy reliance on internal knowledge would also make the arrangement and presentation of the evidence matter less, which weakens the paper’s claim that evidence layout is relatively unimportant for scientific ideation.

This limitation is made worse by the narrow scope of the evaluation metrics. Although the paper runs experiments on two benchmarks with several model variants, its metrics capture only limited aspects of the generated ideas. The ResearchBench matching scores measure agreement with a single reference hypothesis. The CHIMERA study considers only explicit cross-branch citations in the research proposal, that is, whether a record from a different branch is cited as the original instructions require. Neither evaluation assesses novelty, feasibility, or other scientific qualities, or whether the generated ideas differ in substance. Together, these issues restrict how far the findings generalize and weaken the validity, informativeness, and scope of the paper’s main claims.

Questions for authors
  1. In addition to the citation-based metrics, could the authors report content-based evaluation results? Specifically, for the CHIMERA study, do the generated research proposals contain the held-out recombination partners from the original CHIMERA knowledge base?

  2. The paper is motivated by the limited understanding, in prior work, of how knowledge structures affect performance. Given this, why did the authors choose to analyze simple entity-based graph systems and hybrid retrieval strategies, which were not used in those prior works?

6 comments

Thank you very much for taking the time and effort to review our paper, and for your valuable comments and guidance. Your two main concerns helped us make the paper's evidence more complete, and we have run new experiments for each of them. We respond point by point below.

W1: Realism of the structures, comparison methods, and choice of baselines

To test the realism of the structures directly, we added a cross-paper relation experiment. Across 240 tasks, we extracted 1,200 relations, each connecting two papers, and presented them grouped by the connected components of the relation graph. Compared with a flat list, neither LLM judge detected a difference, and adding these relations to the paper records changed scores by only 0.05 and 0.03. This shows that relations have little effect, and that this is not because the relations are confined to single papers. The paper compares four retrieval methods in total, including query-free sequential admission and graph-hop with dense backfill. We chose these baselines to separately test the roles of literature selection and relation presentation while holding the generator and resources fixed. A direct comparison with complete GraphRAG or HippoRAG systems would simultaneously change how evidence is constructed and selected.

W2a: Influence of the generators' internal knowledge, knowledge cutoffs, and publication years

Internal knowledge can contribute to the answers, but external evidence still changes what is generated. We designed a new experiment and re-tested this on 240 tasks with 3 generators and 2 judges (Table R2a). Consistent with the paper, the models do rely on the content of the evidence: replacing the retrieved evidence with irrelevant evidence of the same length lowered scores by 0.19–0.43. We also split the tasks into two groups by how strongly the model depends on the evidence, and found that the effect of presentation is small in both groups. This shows that the conclusion is not an artifact of the models relying on internal knowledge. Regarding publication years, both ResearchBench and CHIMERA use papers published from 2024 onward. Because the generators do not disclose their knowledge cutoffs, training exposure cannot be ruled out from dates alone, which is exactly why we chose to test evidence dependence directly.

Table R2a. Generators' dependence on evidence content

ComparisonScore diff. (min)Score diff. (max)Interpretation
Retrieved − length-matched irrelevant evidence+0.19+0.43Irrelevant content lowers scores
Inspiration papers only − no evidence+0.19+0.45True inspiration sources consistently help
Presentation effect: evidence-dependent vs. evidence-independent tasks−0.02+0.05Unaffected by internal knowledge

W2b: Do the ideas actually change, and what is their quality?

We added a human evaluation of 100 proposal pairs by two raters (with large disagreements re-checked by a third rater), examining both content changes and four aspects of research quality. The two raters jointly found that in the presentation comparisons only details of the content change, whereas in 25/30 access pairs and 28/30 recombination-vs.-refinement pairs the ideas changed substantively, mostly in the method design.

Q1: Content-based evaluation, and whether proposals contain the held-out partner

Thank you for the suggestion. We added three content-level human evaluations, all conducted by two raters who were blind to the source of each proposal (Table R2b; large disagreements were re-checked by a third rater). In the blind quality comparison, changing the presentation does not affect idea quality, recombination is better on all four dimensions, and in 28 of 30 pairs the recombination and refinement proposals indeed differ in substance. In the mechanism check, all 72 proposals that cite literature from other branches substantively use a mechanism from it, whereas none of the 24 non-citing proposals do. Regarding the held-out partner you specifically asked about, it appears in only 2.3–5.4% of proposals, and recombination does not increase this rate. This indicates that the models are not recalling the answer of the original paper, but are using the literature shown to them.

Q2: Why simple entity graphs and hybrid retrieval?

Thank you for the question. Our goal was to make the organization method the only factor that varies. Hybrid retrieval is the most common way ideation systems select papers, and entity linking is the simplest form of cross-paper organization used in prior work; it also generates no summaries, so every piece of evidence placed in the model's context can be precisely controlled and recorded. A complete graph system changes graph construction, retrieval, and summarization at the same time, which makes it hard to tell which component is responsible.

Table R2b. Content-level human evaluation

EvaluationResult
Grouped vs. linear presentationNo difference on any of the four dimensions
Hybrid retrieval vs. graph-hopBetter on three dimensions
Recombination vs. refinementBetter on all four dimensions
Mechanism-use checkCiting: 72/72 substantive use; non-citing: 0/24
Held-out partnerIn only 2.3–5.4% of proposals; no increase under recombination

Presentation: The paper is dense and hard to read

Thank you for candidly pointing this out; we also recognize that the original text is too dense. In the revision, each section of the main text will state its conclusion first, and statistical details will be moved to the appendix to make the paper easier to follow.

Thank you again for your careful and in-depth comments. We have directly tested both of your concerns, cross-paper structure and internal knowledge, and the core conclusions remain unchanged; the new content-level human evaluations are also consistent with the original metrics. We hope these additions address your concerns.

Thank you for the rebuttal. Several of my initial concerns and questions have been resolved, but some remain. My main concern is that the paper still lacks a detailed statistical analysis of the human raters’ content-level evaluations, such as inter-rater agreement. This analysis matters for showing that the evaluation suite is fair. The description of how the human evaluation was conducted, and of which criteria or metrics were used, also remains vague. I have raised my presentation score, but given these remaining concerns, I will maintain my overall score.

Reviewer_Qj9hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors wanted to know whether an "AI scientist" comes up with better research ideas because of "which" papers it is shown or "how" those papers are laid out. They built a controlled test that keeps the two apart: one setup changes which evidence reaches the model under a fixed size limit, and the other arranges the same evidence in different ways (a plain list versus grouped by relationship type). Across more than 68,000 generated ideas, they found that which evidence is delivered matters most: larger pools of papers dilute the useful ones, and one retrieval method scored worse mainly because it handed the model too little text. Rearranging the same evidence made almost no practical difference. In a multi-step version, simply instructing the model to "recombine" ideas raised how often it cited knowledge from other research areas by 52 to 62 percentage points, while the two retrieval methods behaved about the same.

Critical strengths
  1. The paper is well-written on a well motivated problem. The authors asks a clear question and tests it fairly.
  2. The statistics are careful. To claim that "layout does not matter", the authors tested for “practically equivalent” results against a pre-set margin instead of only reporting “no significant difference”. They also used paired comparisons, corrected for multiple tests, and repeated the work with three models and two judges. The authors have been rigorous for their claims.
  3. The authors were honest about their limitations and investigated the reasons "why" a result happened. They performed extra checks, such as filling unused space with more papers, showed that the weaker retrieval method failed because it gave the model too little text, and that fixing this closed the gap. The authors also openly report which decisions were made after seeing results and that their quality judges failed validation, so they switched to a measure that needs no judge.
Critical weaknesses
  1. The central finding is fairly obvious. What the model sees matters more than how it is arranged means that good retrieval leads to better results which is not surprising.
  2. Only one kind of layout was tested. Would the “layout does not matter” result hold for richer layouts such as citation chains, graphs, or summaries?
  3. The main score measures overlap with a reference hypothesis, which mostly rewards retrieving the right papers rather than producing novel or useful ideas. How this is a useful proxy?
  4. The automatic judges were small models with modest agreement against humans, checked on only 40 examples from one generator. Reasoning was turned off for all models.
Questions for authors

See above

1 comment

We sincerely thank you for reviewing our paper, for the time and effort you devoted to it, and for your valuable comments and guidance. Your comments were very helpful, and we have run additional experiments accordingly. We respond to each point below.

W1: Is the core finding obvious?

We understand that this conclusion may sound intuitive at first. However, one aspect of our results is not obvious: the gap between graph-based methods and text retrieval is determined almost entirely by how much evidence ultimately reaches the model's context, not by the structure itself. Graph-hop originally used only 27–28% of the context budget; once its unused budget was filled, it instead scored 0.10–0.15 higher than hybrid retrieval (Table 7 in the paper). Conversely, when hybrid retrieval was truncated to the same amount of evidence as graph-hop, the gap between the two also disappeared. Previous system comparisons typically did not control for this, so apparent structural advantages or disadvantages may simply reflect differences in how much of the budget was used.

W2: Only one layout was tested. Does the "layout does not matter" result hold for richer layouts such as citation chains, graphs, or summaries?

We have added a graph-layout experiment. Across 240 tasks, we extracted 1,200 relations, each connecting two papers, and presented them either as a linear list or grouped by the connected components of the relation graph. Two human raters audited a sample of 100 of these relations (with large disagreements re-checked by a third rater) and judged 0.97 and 0.90 of them valid. Over the 1,200 relations, neither LLM judge detected a difference between the two layouts, consistent with the paper's conclusion. In the revision, we will state the scope within which this conclusion has been tested for cross-paper relations. As for summary-based layouts, they rewrite the evidence itself, which amounts to changing what the model sees; this is precisely the type of factor that our paper finds to affect outcomes.

W3: Is the matching score a useful proxy?

Thank you for raising this; it is an issue we take seriously. The matching score measures whether the model can recover the research hypothesis of the original paper from the given evidence, so it directly reflects whether the evidence is actually used. In a new experiment, we replaced the retrieved evidence with evidence of the same length but low relevance, and the matching score dropped by 0.19–0.43, consistent with the paper's conclusion. To evaluate idea quality directly, we also had two raters blindly compare 100 pairs of proposals (with large disagreements re-checked by a third rater). The results point in the same direction as the matching score: changing the presentation does not affect novelty, soundness, feasibility, or scientific value.

W4: Scale and reliability of the LLM judges, and disabled reasoning

Thank you for pointing out these issues in detail. The two LLM judges are mixture-of-experts models with 125B and 552B parameters, respectively. To address validation on a single generator, we added human scoring (with large disagreements re-checked by a third rater) of 72 outputs covering all three generators, and the LLM judges' rankings agree with the human ratings. The LLM judges score about 0.37 points higher than humans overall (Table R1a), but all comparisons are paired within the same generator, so this fixed offset cancels out. Our original intent in disabling reasoning was to give every call the same output budget, so that conditions differ only in their evidence. We have now added a reasoning on/off experiment: reasoning raises scores overall, but both core conclusions still hold.

Table R1a. Results with reasoning enabled, and cross-generator validation of the judges

CheckQwen-Flash judgeDeepSeek-Flash judge
Access gap, reasoning on (hybrid − graph-hop)+0.288+0.202
Presentation gap, reasoning on (linear − relation-grouped)−0.028−0.023
Rank correlation with the two raters0.70 / 0.670.62 / 0.62
Mean score offset vs. the two raters+0.39 / +0.38+0.38 / +0.36

Thank you again for the time you have invested in our work and for your valuable comments. The experiments they led to have made the paper’s conclusions more solid.

Comments on the paper

2 comments

Dear Reviewers, thank you for your thoughtful reviews. As the authors have now responded, please review their responses and the other reviews, and leave follow-up comments on whether the concerns have been addressed. Thank you for your time and effort.

AuthorsOpenReview ↗

Final Response and Appreciation

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely thank you for the valuable time and effort you have devoted to our paper, and for organizing and overseeing the review and discussion with such care and responsibility. We are also deeply grateful to all four reviewers for the time and effort they devoted to our paper, and for their careful reviews and valuable comments. We especially appreciate their recognition that our work investigates an important yet underexplored question in automated scientific ideation, offers a new perspective by separating how the knowledge that influences ideation is accessed from how it is presented, and holds practical significance and methodological value for guiding the design of future ideation systems, supported by a controlled experimental design and rigorous statistical analysis. During the rebuttal, we added human evaluations of idea quality and further analyses of cross-paper structures, the generators’ internal knowledge, evidence quantity, and reasoning, all of which support our core conclusions. We respectfully hope that the final assessment will consider the paper’s core contributions, as well as the additional analyses and clarifications provided during the rebuttal.

Thank you for your hard work and support.

Best regards,

The authors of Paper 60511

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.