- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FlowPIE, a scientific idea generation framework that couples literature retrieval with idea generation. First, an MCTS search over a literature graph explores trajectories of papers. Its node selection is guided by GRM rewards on the ideas each trajectory produces, and the generated ideas form an initial population. That population is then evolved at test time. Crossover combines parent ideas, and mutation brings in literature from distant parts of the graph (“isolation islands”). Fitness comes from an LLM-based generative reward model (GRM). Experiments on AI Idea Bench 2025 and IdeaBench show FlowPIE outperforming several LLM-based and agent-based baselines, and a small-scale human evaluation supports the same result.
Critical strengths
- Well-motivated problem. The paper argues that retrieval should adapt to the quality of the ideas being generated rather than be a fixed preprocessing step. This is reasonable and timely, and RQ1/RQ2 are stated clearly.
- Coherent system design. The two stages have clearly separated jobs: one builds a high-quality initial population, the other refines it. The algorithms are described fully, and the prompts are included.
- Broad experiments. the experiments cover a fair amount of ground (two benchmarks, several domains, a few backbones, a cost table).
Critical weaknesses
- The optimization target is also used for evaluation, so reward hacking is a risk. The GRM is the fitness function during evolution. Of course FlowPIE wins there. The authors should show that the gains hold under evaluation independent of the GRM, and explain how the method avoids overfitting to it.
- I’m worried about data leakage. The main experiments retrieve from USPTO patents published in late 2025, and patents often appear well after the work they describe. It’s quite possible some of them overlap with the benchmark’s target papers. A per-query check of the retrieved documents against the target paper would be much more convincing.
- Some numbers don’t line up across tables. In Table 1, FlowPIE (patent) gets I2T/I2I/IMCQ of 4.64/4.44/0.780. In Table 6, the evolved FlowPIE gets 4.18/4.15/0.708. Research Agent’s I2T is 4.56 in Table 1 and 4.08 in Table 10. As far as I can tell these use the same backbone and benchmark, so I don’t understand where the differences come from. This needs to be cleared up, since it affects how much I trust the main table.
- The compute comparison isn’t fair. FlowPIE makes many MCTS rollouts, calls the GRM repeatedly, and runs up to 20 generations. The baselines generate more or less once. Table 8 gives cost per idea, but not the total cost to get the final three ideas. I also couldn’t find how those three ideas are chosen from the final population. Moreover, the paper misses obviou baseline: give the existing methods the same token budget, sample repeatedly, and pick with the same GRM. A comparison with existing LLM-based evolutionary methods would also help.
Questions for authors
- What explains the score differences between Tables 1, 6 and 10?
- How does FlowPIE compare with the baselines plus GRM best-of-N at a matched budget?
- How much of the evolution gain survives with an independent judge?
4 comments
Response by Authors (Part 2)
Dear Reviewer Xy7E,
Weakness 3 & Question 1: Numerical Discrepancies Across Tables 1, 6, and 10
We sincerely thank you for carefully examining the numerical results. We would like to kindly clarify that the reported values are correct. However, we apologize for not clearly stating that the latter represents an aggregated score rather than an independent metric.
Specifically, several evaluation dimensions, such as I2T, consist of two sub-metrics(Motivation and Exp Plan). Table 1 reports the score obtained from these sub-metrics, whereas Tables 6 and 10 report the average score of a single corresponding sub-metric.
For example, the results of FlowPIE (patent) are presented below:
| Metric | Motivation | Exp. Plan | Aggregated Score (Table 6) |
|---|---|---|---|
| I2T | 4.64 | 3.72 | 4.18 |
| I2I | 4.44 | 3.85 | 4.15 |
| IMCQ | 0.780 | 0.635 | 0.708 |
The values reported in Table 6 are obtained by averaging the corresponding Motivation and Exp. Plan scores. Thus, although the tables use similar metric labels, they report results at different levels of aggregation.
To eliminate this ambiguity, we will revise the table captions and metric descriptions to explicitly specify the underlying sub-metrics and aggregation formulas.
Weakness 4 & Question 2: Computational Fairness and Budget-Matched Comparison
Answer: We sincerely thank you for the constructive suggestions regarding computational fairness and budget-matched comparisons.
First, we would like to clarify that several baseline methods are not single-pass generation frameworks. For instance, ResearchAgent incorporates iterative refinement, VirSci employs multi-agent collaboration, and other frameworks involve multiple rounds of retrieval, generation, or evaluation. Therefore, FlowPIE is not the only approach that incurs additional inference costs through multi-stage reasoning and refinement.
However, strictly enforcing identical token or API budgets across scientific idea generation frameworks is inherently challenging, since these methods differ substantially in their underlying architectures, including the number of agents (VirSci), iterative refinement procedures (ResearchAgent), retrieval steps (Chian-of-Ideas), and search strategies (SCIPIP).
To ensure a consistent and reproducible comparison, we adopt the same backbone LLM across all methods and follow the official implementations and recommended hyperparameter configurations of the baselines. Furthermore, the computational costs reported in Table 8 (Page 14) include all generation and reward-model calls throughout FlowPIE's complete search and evolution process, rather than only the final generation stage.
Importantly, FlowPIE does not incur the highest API cost among the compared frameworks, while methods such as Chain-of-Ideas and VirSci exhibit comparable or higher costs per idea.
Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.
Best Regards,
Authors of #6495
Sincerely Looking Forward to Your Response
Dear Reviewer Xy7E,
We hope this message finds you well. We understand that you may have a busy schedule, so we would like to kindly remind you about our responses to your comments. We sincerely appreciate the time and effort you have taken to provide valuable feedback on our manuscript. We deeply value your feedback and believe that our responses address the concerns you raised.
As the discussion phase will close in two days, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.
We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.
Best Regards,
Authors of #6495
Sincerely Looking Forward to Your Response (One Day Remaining)
Dear Reviewer Xy7E,
We hope this message finds you well. As the discussion phase will close in one day, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.
We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.
Best Regards,
Authors of #6495
Response by Authors (Part 1)
Dear Reviewer Xy7E,
We sincerely thank the reviewer for the careful evaluation of our work and for the constructive and insightful feedback. We are grateful for your positive recognition of the problem settings, coherent system design and broad experiments.
We particularly appreciate the concerns regarding evaluation independence, potential data leakage, metric consistency, and computational fairness. We would like to provide clarification on your concerns, and kindly request your thoughtful reconsideration of our manuscript.
Answer: We sincerely thank you for highlighting the potential risk of reward hacking arising from using the GRM for both evolutionary optimization and evaluation.
We conduct additional evaluations using two independent LLM judges, GPT-5-mini and Gemini-3.1-Flash-Lite, alongside the original DeepSeek-V3.2 evaluator. The additional two judges were not involved in the evolutionary optimization or candidate selection process.
The results consistently demonstrate improvements from the initial population to the final FlowPIE outputs:
Notably, FlowPIE consistently achieves the best performance across different GRMs, suggesting that the effectiveness of our method does not rely on a specific reward model. Additionally, we report a high inter-model agreement among the three evaluators, with Kendall’s W=0.8651.
Regarding Question 3, the evolution-related improvement is preserved under both independent judges, with average score gains of +0.06 under GPT-5-mini and +0.03 under Gemini-3.1-Flash-Lite, compared with the initial population.
Answer: We sincerely thank you for raising this important concern regarding potential data leakage between our literature corpus and the benchmark target papers. We would like to clarify that we have already conducted a data quality analysis to investigate potential knowledge leakage, as reported in Appendix B (Data Quality Analysis). Following Si et al. (2025), we encode the literature corpus, including both scientific papers and patents, using all-MiniLM-L6-v2 and compute cosine similarities against the target papers in both benchmarks.
As shown in Figure 5 (Page 14), most similarity scores fall below 0.3, and no literature documents approach the predefined duplication threshold of 0.65. These results suggest limited semantic overlap between our literature database and the benchmark target papers , poses minimal risk of data leakage.