acceptodds
Under review as a conference paper at ICLR 2027

FlowPIE: Test-Time Scientific Idea Evolution with Flow-Guided Literature Exploration

Abstract

Scientific idea generation (SIG) is critical to AI-driven autonomous research, yet existing approaches are often constrained by a static retrieval-then-generation paradigm, leading to homogeneous and insufficiently divergent ideas. In this work, we propose FlowPIE, a tightly coupled retrieval–generation framework that treats literature exploration and idea generation as a co-evolving process. FlowPIE expands literature trajectories via a flow-guided Monte Carlo Tree Search (MCTS) inspired by GFlowNets, using the quality of current ideas assessed by an LLM-based generative reward model (GRM) as a supervised signal to guide adaptive retrieval and construct a diverse, high-quality initial idea population. Based on this population, FlowPIE models idea generation as a test-time idea evolution process, applying selection, crossover, and mutation with the isolation island paradigm and GRM-based fitness computation to incorporate cross-domain knowledge. It effectively mitigates the information cocoons arising from over-reliance on parametric knowledge and static literature. Extensive evaluations demonstrate that FlowPIE consistently produces ideas with higher novelty, feasibility and diversity compared to strong LLM-based and agent-based frameworks, while enabling reward scaling during test time. Our code and datasets will be released to facilitate future research on autonomous scientific ideation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Models, Idea Generation, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xy7EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FlowPIE, a scientific idea generation framework that couples literature retrieval with idea generation. First, an MCTS search over a literature graph explores trajectories of papers. Its node selection is guided by GRM rewards on the ideas each trajectory produces, and the generated ideas form an initial population. That population is then evolved at test time. Crossover combines parent ideas, and mutation brings in literature from distant parts of the graph (“isolation islands”). Fitness comes from an LLM-based generative reward model (GRM). Experiments on AI Idea Bench 2025 and IdeaBench show FlowPIE outperforming several LLM-based and agent-based baselines, and a small-scale human evaluation supports the same result.

Critical strengths
  1. Well-motivated problem. The paper argues that retrieval should adapt to the quality of the ideas being generated rather than be a fixed preprocessing step. This is reasonable and timely, and RQ1/RQ2 are stated clearly.
  2. Coherent system design. The two stages have clearly separated jobs: one builds a high-quality initial population, the other refines it. The algorithms are described fully, and the prompts are included.
  3. Broad experiments. the experiments cover a fair amount of ground (two benchmarks, several domains, a few backbones, a cost table).
Critical weaknesses
  1. The optimization target is also used for evaluation, so reward hacking is a risk. The GRM is the fitness function during evolution. Of course FlowPIE wins there. The authors should show that the gains hold under evaluation independent of the GRM, and explain how the method avoids overfitting to it.
  2. I’m worried about data leakage. The main experiments retrieve from USPTO patents published in late 2025, and patents often appear well after the work they describe. It’s quite possible some of them overlap with the benchmark’s target papers. A per-query check of the retrieved documents against the target paper would be much more convincing.
  3. Some numbers don’t line up across tables. In Table 1, FlowPIE (patent) gets I2T/I2I/IMCQ of 4.64/4.44/0.780. In Table 6, the evolved FlowPIE gets 4.18/4.15/0.708. Research Agent’s I2T is 4.56 in Table 1 and 4.08 in Table 10. As far as I can tell these use the same backbone and benchmark, so I don’t understand where the differences come from. This needs to be cleared up, since it affects how much I trust the main table.
  4. The compute comparison isn’t fair. FlowPIE makes many MCTS rollouts, calls the GRM repeatedly, and runs up to 20 generations. The baselines generate more or less once. Table 8 gives cost per idea, but not the total cost to get the final three ideas. I also couldn’t find how those three ideas are chosen from the final population. Moreover, the paper misses obviou baseline: give the existing methods the same token budget, sample repeatedly, and pick with the same GRM. A comparison with existing LLM-based evolutionary methods would also help.
Questions for authors
  1. What explains the score differences between Tables 1, 6 and 10?
  2. How does FlowPIE compare with the baselines plus GRM best-of-N at a matched budget?
  3. How much of the evolution gain survives with an independent judge?
4 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

Dear Reviewer Xy7E,

We sincerely thank the reviewer for the careful evaluation of our work and for the constructive and insightful feedback. We are grateful for your positive recognition of the problem settings, coherent system design and broad experiments.

We particularly appreciate the concerns regarding evaluation independence, potential data leakage, metric consistency, and computational fairness. We would like to provide clarification on your concerns, and kindly request your thoughtful reconsideration of our manuscript.

Weakness 1 & Question 3: Reward Hacking and Independent Evaluation

Answer: We sincerely thank you for highlighting the potential risk of reward hacking arising from using the GRM for both evolutionary optimization and evaluation.

We conduct additional evaluations using two independent LLM judges, GPT-5-mini and Gemini-3.1-Flash-Lite, alongside the original DeepSeek-V3.2 evaluator. The additional two judges were not involved in the evolutionary optimization or candidate selection process.

The results consistently demonstrate improvements from the initial population to the final FlowPIE outputs:

MethodDSGPTGemini
NFAvg.NFAvg.NFAvg.
SCIPIP0.500.730.620.370.710.540.380.760.57
Research Agent0.490.660.580.390.750.570.420.730.58
Chain-of-Ideas0.590.660.630.520.730.630.540.750.65
VirSci0.600.650.630.400.670.540.510.700.61
FlowPIE (Ours)0.750.760.760.520.750.640.590.790.69
Initial Population0.680.680.680.410.750.580.520.800.66

Notably, FlowPIE consistently achieves the best performance across different GRMs, suggesting that the effectiveness of our method does not rely on a specific reward model. Additionally, we report a high inter-model agreement among the three evaluators, with Kendall’s W=0.8651.

Regarding Question 3, the evolution-related improvement is preserved under both independent judges, with average score gains of +0.06 under GPT-5-mini and +0.03 under Gemini-3.1-Flash-Lite, compared with the initial population.

Weakness 2: Potential Data Leakage in Patent Retrieval

Answer: We sincerely thank you for raising this important concern regarding potential data leakage between our literature corpus and the benchmark target papers. We would like to clarify that we have already conducted a data quality analysis to investigate potential knowledge leakage, as reported in Appendix B (Data Quality Analysis). Following Si et al. (2025), we encode the literature corpus, including both scientific papers and patents, using all-MiniLM-L6-v2 and compute cosine similarities against the target papers in both benchmarks.

As shown in Figure 5 (Page 14), most similarity scores fall below 0.3, and no literature documents approach the predefined duplication threshold of 0.65. These results suggest limited semantic overlap between our literature database and the benchmark target papers , poses minimal risk of data leakage.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Dear Reviewer Xy7E,

Weakness 3 & Question 1: Numerical Discrepancies Across Tables 1, 6, and 10

We sincerely thank you for carefully examining the numerical results. We would like to kindly clarify that the reported values are correct. However, we apologize for not clearly stating that the latter represents an aggregated score rather than an independent metric.

Specifically, several evaluation dimensions, such as I2T, consist of two sub-metrics(Motivation and Exp Plan). Table 1 reports the score obtained from these sub-metrics, whereas Tables 6 and 10 report the average score of a single corresponding sub-metric.

For example, the results of FlowPIE (patent) are presented below:

MetricMotivationExp. PlanAggregated Score (Table 6)
I2T4.643.724.18
I2I4.443.854.15
IMCQ0.7800.6350.708

The values reported in Table 6 are obtained by averaging the corresponding Motivation and Exp. Plan scores. Thus, although the tables use similar metric labels, they report results at different levels of aggregation.

To eliminate this ambiguity, we will revise the table captions and metric descriptions to explicitly specify the underlying sub-metrics and aggregation formulas.

Weakness 4 & Question 2: Computational Fairness and Budget-Matched Comparison

Answer: We sincerely thank you for the constructive suggestions regarding computational fairness and budget-matched comparisons.

First, we would like to clarify that several baseline methods are not single-pass generation frameworks. For instance, ResearchAgent incorporates iterative refinement, VirSci employs multi-agent collaboration, and other frameworks involve multiple rounds of retrieval, generation, or evaluation. Therefore, FlowPIE is not the only approach that incurs additional inference costs through multi-stage reasoning and refinement.

However, strictly enforcing identical token or API budgets across scientific idea generation frameworks is inherently challenging, since these methods differ substantially in their underlying architectures, including the number of agents (VirSci), iterative refinement procedures (ResearchAgent), retrieval steps (Chian-of-Ideas), and search strategies (SCIPIP).

To ensure a consistent and reproducible comparison, we adopt the same backbone LLM across all methods and follow the official implementations and recommended hyperparameter configurations of the baselines. Furthermore, the computational costs reported in Table 8 (Page 14) include all generation and reward-model calls throughout FlowPIE's complete search and evolution process, rather than only the final generation stage.

Importantly, FlowPIE does not incur the highest API cost among the compared frameworks, while methods such as Chain-of-Ideas and VirSci exhibit comparable or higher costs per idea.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #6495

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response

Dear Reviewer Xy7E,

We hope this message finds you well. We understand that you may have a busy schedule, so we would like to kindly remind you about our responses to your comments. We sincerely appreciate the time and effort you have taken to provide valuable feedback on our manuscript. We deeply value your feedback and believe that our responses address the concerns you raised.

As the discussion phase will close in two days, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #6495

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response (One Day Remaining)

Dear Reviewer Xy7E,

We hope this message finds you well. As the discussion phase will close in one day, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #6495

Reviewer_H999OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the conventional scientific idea generation paradigm, in which literature is statically retrieved before idea generation, and proposes FlowPIE, a novel framework for test-time idea evolution. FlowPIE integrates initial idea generation with dynamic literature exploration through flow-guided MCTS, while continuously refining ideas using evolutionary operators, including selection, crossover, and mutation.

Critical strengths

Introduce FlowPIE, a tightly coupled retrieval–generation framework that reformulates scientific idea generation as a test-time idea evolution process propose a flow-guided MCTS mechanism for adaptive literature exploration and design an isolation-island-enhanced mutation strategy

Critical weaknesses

The proposed framework mainly combines existing techniques, including MCTS, evolutionary algorithms, and LLM-based reward evaluation. The flow-guided component appears to be a reward-based modification of MCTS, and its differences from standard MCTS are not sufficiently established.

The paper does not compare flow-guided MCTS against standard MCTS or simpler adaptive retrieval strategies. It also lacks separate ablations for crossover and mutation. As a result, it is difficult to identify which components are responsible for the improvements.

Questions for authors

FlowPIE does not consistently outperform the baselines on IdeaBench. For example, Chain-of-Ideas achieves higher novelty, while Research Agent achieves higher feasibility. The human evaluation also shows only modest improvements over SCIPIP.

The same GRM is used to guide idea evolution and demonstrate test-time reward scaling. Increasing reward scores under this setup does not necessarily indicate better research ideas. Stronger evidence from independent evaluation across evolution steps would make the claims more convincing.

Prior methods such as LDC: Learning to Generate Research Idea with Dynamic Control have explored improving research idea generation through multi-dimensional reward feedback. Although FlowPIE focuses on test-time optimization, its relationship to existing reward-guided idea generation methods deserves further discussion.

4 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

Dear Reviewer H999,

We sincerely thank you for the careful evaluation of our work and for the constructive and insightful feedback. We particularly appreciate the concerns regarding evaluation independence, potential data leakage, metric consistency, and computational fairness. We would like to provide clarification on your concerns, and kindly request your thoughtful reconsideration of our manuscript.

Weaknesses 1 & 2: Methodology and Component-Level Contributions

Answer: Thank you for considering the contribution of our methodology. We acknowledge that our framework builds upon established techniques. However, inspired by GFlowNet, we propose Flow-Guided MCTS to facilitate more effective and diverse literature exploration, together with population-based idea evolution to progressively improve the quality and diversity of generated research ideas.

  • (1) Contribution of Flow-Guided MCTS. To distinguish our approach from standard MCTS, we conduct an additional controlled ablation study comparing Flow-Guided MCTS with Standard MCTS, as shown in Table below. | Method | Diversity | | :--- | :--- | | FlowPIE (Full) | 0.50 | | w/o Evolution (Flow-Guided MCTS) | 0.31 | | w/o Evolution (Standard MCTS) | 0.21 |

The results show that Flow-Guided MCTS achieves 45.7% higher diversity than Standard MCTS. This controlled comparison demonstrates that the global flow mechanism influences search behavior beyond conventional reward-guided MCTS, promoting broader exploration of the literature space.

  • (2) Contribution of Idea Evolution. Beyond the search process, we evaluate the effectiveness of idea evolution by using ideas generated by different methods as initial populations and applying the same evolutionary operations, as shown in Table 6 (Page 9).

These results demonstrate that, when ideas generated by different methods are used as the initial populations, the evolutionary operations consistently different improvements in idea quality to varying degrees.

Question 1: Performance on IdeaBench and Human Evaluation

Answer: We sincerely thank you for the careful examination of our experimental results. We acknowledge that FlowPIE does not achieve the highest score on every individual metric, as different methods exhibit strengths in different aspects of idea quality.

However, FlowPIE demonstrates strong and competitive performance across multiple IdeaBench metrics (Table 2, Page 7). Specifically, FlowPIE achieves the highest Semantic Similarity (0.559) and Idea Overlap (7.76), while ranking second in Novelty Insight (0.825), behind Chain-of-Ideas (0.926). Although its Feasibility Insight score (0.105) is below those of Research Agent (0.138) and SCIPIP (0.133), it remains higher than Chain-of-Ideas (0.095) and VirSci (0.075).

These results indicate that FlowPIE achieves competitive performance across multiple dimensions of idea quality, rather than optimizing for a single metric. We also acknowledge the novelty–feasibility trade-off and will clarify that our objective is to improve overall idea quality through structured literature exploration and evolution, rather than guarantee the highest score on every individual evaluation dimension. Furthermore, our human evaluation (Table 3,Page7) provides complementary evidence of FlowPIE's effectiveness.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Dear Reviewer H999,

Question 2: Independent Evaluation of Evolutionary Improvements

Answer: We sincerely thank you for your suggestion on evaluation setting. We conduct additional evaluations using two independent LLM judges, GPT-5-mini and Gemini-3.1-Flash-Lite, alongside the original DeepSeek-V3.2 evaluator. The independent evaluators were not involved in the evolutionary optimization or candidate selection process.

MethodDSGPTGemini
NFAvg.NFAvg.NFAvg.
SCIPIP0.500.730.620.370.710.540.380.760.57
Research Agent0.490.660.580.390.750.570.420.730.58
Chain-of-Ideas0.590.660.630.520.730.630.540.750.65
VirSci0.600.650.630.400.670.540.510.700.61
FlowPIE (Ours)0.750.760.760.520.750.640.590.790.69
Initial Population0.680.680.680.410.750.580.520.800.66

As result shown, FlowPIE consistently achieves the best performance across different evaluators, suggesting that the effectiveness of our method does not rely on a specific reward model. Additionally, we report a high inter-model agreement among the three evaluators, with Kendall’s W=0.8651.

Question 3: Relationship to LDC and Prior Reward-Guided Ideation Methods

Answer: We sincerely thank you for highlighting this relevant work. We agree that LDC and FlowPIE both leverage reward feedback to improve scientific idea generation, but they differ in their optimization strategies.

LDC primarily employs supervised fine-tuning and reinforcement learning to improve the idea generator, with dynamic control at inference time. In contrast, FlowPIE focuses on test-time optimization, combining flow-guided literature exploration with population-based idea evolution to improve idea quality without updating the backbone LLM.

We view these approaches as complementary and will further clarify their differences and discuss LDC in the revised manuscript.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #6495

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response

Dear Reviewer H999,

We hope this message finds you well. We understand that you may have a busy schedule, so we would like to kindly remind you about our responses to your comments. We sincerely appreciate the time and effort you have taken to provide valuable feedback on our manuscript. We deeply value your feedback and believe that our responses address the concerns you raised.

As the discussion phase will close in two days, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #6495

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response (One Day Remaining)

Dear Reviewer H999,

We hope this message finds you well. As the discussion phase will close in one day, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #6495

Reviewer_vLREOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlowPIE is a two-stage framework for generating and refining scientific research ideas. First, GFlowNet-inspired MCTS explores a literature graph and uses idea-quality feedback to construct an initial population. Second, crossover, mutation, and selection refine the population with additional “isolation island” literature from distant graph regions. An LLM-based reward model assesses novelty and feasibility throughout the process. The main contribution is the integration of adaptive literature exploration with test-time idea evolution. Validation includes two benchmarks, human evaluation, different generators and literature sources, and ablation studies.

Critical strengths
  1. The framework offers a well-motivated approach to adaptive literature exploration. Topically relevant papers do not necessarily provide useful inspiration. FlowPIE uses feedback on generated ideas to adjust exploration, then refines candidates with additional knowledge. This directly addresses the gap between retrieving relevant literature and developing research ideas. The integrated design is a meaningful conceptual contribution to scientific idea generation.
  2. The evaluation provides encouraging evidence for the framework’s effectiveness. Table 1 reports gains in IMCQ, while Table 10 extends evaluation to additional generators. A separate benchmark judge and human assessment provide evidence beyond the optimization reward. These results strengthen the empirical case for the overall framework, particularly on the dimensions where improvements are observed.
Critical weaknesses
  1. The evidence does not isolate the contribution of global flow. Algorithm 1 updates normalized transition probabilities and then propagates flow along the tree. This propagation does not itself establish an additional decision constraint. Table 6 replaces initial populations rather than comparing search mechanisms under matched conditions. The results therefore leave unclear what the flow formulation adds beyond reward-guided tree search. This limits the methodological novelty that can be attributed to this component.
  2. The quality claims exceed the evidence for joint improvement. In Table 2, evolution improves novelty but reduces Feasibility Insight from 0.136 to 0.105. The final feasibility score is also below SCIPIP and Research Agent. A novelty–feasibility trade-off does not establish improvement in both objectives. Since both are central evaluation criteria, this narrows the demonstrated benefit of evolution and calls for more qualified conclusions.
Questions for authors
  1. Would removing global-flow computation, while leaving the other update rules unchanged, alter search behavior?
  2. Is isolation island literature supplied to crossover, mutation, or both? Section 3.2 and Algorithm 2 introduce it through probabilistic mutation. However, Figure 9 already includes it in the crossover prompt.
  3. Does Table 1 average the scores of all three ideas within each query, or take the maximum before averaging across queries?
  4. Was July 2024 the actual cutoff for human novelty assessment? The retrieval corpus includes patents published in October–December 2025. If evaluators were not required to treat this later knowledge as prior work, the human scores would not establish novelty relative to the literature available to the system.
  5. Are the correlations in Table 5 computed across individual ideas or across method-level mean scores?
  6. Does the cost reported in Table 8 include all generation and reward-model calls during search and evolution?
4 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

Dear Reviewer vLRE,

We sincerely appreciate the time and effort you have invested in reviewing our paper and providing constructive feedback. We are grateful for your positive recognition of the presentation of our paper, the motivation of our approach and the effectiveness of our frameworks. We would like to provide clarification on your concerns, and kindly request your thoughtful reconsideration of our manuscript.

Weakness 1 & Question 1: Contribution of Global Flow

Answer: We appreciate your thoughtful consideration of the contribution of global flow. To further clarify the independent contribution of global flow, we conducted an additional ablation study by removing the global flow computation, i.e., standard MCTS, and replace it with Flow-Guided MCTS , while keeping all other settings unchanged. The results are summarized below:

MethodDiversity
FlowPIE (Full)0.50
w/o Isolation Island0.43
w/o Evolution (Flow-Guided MCTS)0.31
w/o Evolution (Standard MCTS)0.21

The results show that removing global flow reduces diversity from 0.31 to 0.21, a decrease of 31.4%. This difference provides direct evidence that global flow influences the search policy and promotes more diverse exploration. Without global flow, Standard MCTS tends to concentrate exploration on previously favored literature, resulting in reduced diversity. We will clarify the role of global flow in the methodology into the revised manuscript.

Weakness 2: Effectiveness of Idea Evolution and Novelty–Feasibility Trade off

Answer: We sincerely thank you for highlighting the effectiveness of idea evolution and the novelty-feasibility trade off. We agree that improving novelty does not necessarily imply simultaneous improvements in feasibility, and acknowledge that Feasibility Insight decreases from 0.136 to 0.105 after evolution.

However, we would like to emphasize that the countribution of idea evolution extends beyond improving single evaluation metric. As discussed in Table2(Page 7), FlowPIE lies on the Pareto frontier of Novety Insight (NI) and Feasibility Insight (FI), indicating a competitive balance between these two objectives rather than superiority in either dimension individually. Moreover, evolution improves Novelty Insight from 0.750 to 0.825, while also increasing Semantic Similarity from 0.532 to 0.559 and Idea Overlap from 7.64 to 7.76. Although Feasibility Insight decreases (0.136 to 0.105), these results demonstrate improvements across multiple evaluation dimensions.

Futhermore, Table 6 (Page 8) validates the effectiveness of evolution across different initial populations. Specifically, when using ideas generated by SCIPIP, Chain-of-Ideas, and FlowPIE as initial populations, applying the same evolutionary operations consistently improves the average reward (by +3.3%, +8.5%, and +10.1%, respectively).

Question 2: Use of Isolation Island Literature in Crossover and Mutation

Answer: We sincerely thank you for pointing out this ambiguity. We would like to clarify that isolation island literature is incorporated into both crossover and mutation, but serves distinct purposes in each operator.

  • In crossover, island literature is provided as auxiliary context to guide the recombination of two parent ideas. As illustrated in the crossover prompt, it helps the LLM integrate complementary technical features while encouraging novelty and diversity. The primary objective remains the synthesis of the two parent ideas.

  • In mutation, the Isolation Island strategy is explicitly activated with probability (ρ\rho). When triggered, literature from topologically distant subgraphs is used to perturb the offspring idea, encouraging exploration beyond the current local neighborhood and helping escape local optima.

We will revise the descriptions of both operators to explicitly clarify their respective roles within the methodology and implementation.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Dear Reviewer vLRE,

Question 3: Score Aggregation in Table 1

Answer: We sincerely thank you for your careful consideration of the score aggregation in Table 1. For each query, we first compute the average score of the three generated ideas. We then calculate the mean and standard deviation of these query-level scores across all queries in the benchmark. The results in Table 1 are reported as mean ± standard deviation (e.g., 4.66±0.599), where the standard deviation reflects the variability in average idea quality across different queries.

Question 4: Temporal Cutoff for Human Novelty Assessment

Answer: We sincerely thank you for raising this important question regarding the temporal consistency between human novelty assessment and patent retrieval. We would like to clarify the motivation behind this temporal cutoff setting. Patent publication dates do not necessarily reflect when the underlying technologies were originally developed. Under standard USPTO procedures, patent applications are generally published approximately 18 months after their earliest effective filing or priority date. Thus, patents published in late 2025 may originate from inventions conceived and filed in early or mid-2024. Furthermore, our choice of the temporal reference point is consistent with the time-alignment approach adopted in the prior work by Si et al. (2025), whose human evaluation protocol we follow.

Question 5: Correlation Computation in Table 5

Answer: We sincerely thank you for your careful consideration of correlation computation. We would like to clarify that the Spearman correlations are computed at the individual idea level, rather than based on method-level mean scores. Specifically, for each generated idea, we collect the corresponding human evaluation scores and reward scores. The Human–Human correlations are computed between individual experts' ratings, while the Human–Reward correlations are computed between the reward scores and the average human ratings for the same ideas.

As reported in Table 5 (Page 7), the Human–Reward correlations are 0.60 for novelty and 0.87 for feasibility, indicating positive agreement between the reward model and aggregated human judgments at the idea level.

Question 6: Computational Cost Accounting

Answer: We sincerely thank you for your detailed consideration of cost accounting. Yes, the API costs reported in Table 8 (Page 14) include all LLM generation and GRM evaluation calls throughout the entire search and evolution processes. Therefore, the reported costs reflect the complete generation and reward-evaluation overhead of FlowPIE.

Furthermore, as shown in Table 8, FlowPIE does not incur the highest API cost among the compared methods. Other frameworks, such as Chain-of-Ideas and VirSci, exhibit comparable or higher costs per idea, despite their different generation architectures. We will explicitly clarify the cost accounting procedure and its scope in the revised manuscript to avoid ambiguity.


Sincerely thank you again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #6495

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response

Dear Reviewer vLRE,

We hope this message finds you well. We understand that you may have a busy schedule, so we would like to kindly remind you about our responses to your comments. We sincerely appreciate the time and effort you have taken to provide valuable feedback on our manuscript. We deeply value your feedback and believe that our responses address the concerns you raised.

As the discussion phase will close in two days, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #6495

AuthorsOpenReview ↗

Sincerely Looking Forward to Your Response (One Day Remaining)

Dear Reviewer vLRE,

We hope this message finds you well. As the discussion phase will close in one day, we would be very grateful to receive any further feedback from you before the deadline. If you find our clarifications satisfactory, we kindly request your thoughtful reconsideration of our submission, and we look forward to your response.

We would also like to kindly remind you that we have uploaded a revised version of our manuscript to address your concerns.

Best Regards,

Authors of #6495

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Uploaded

We sincerely thank all reviewers for your constructive feedback. We have uploaded a revised manuscript with the following key updates:

  • We supplement the experiment on the analysis of global flow impact in the main text at line 227. Details can be found in Appendix C.3. (Reviewer vLRE, Weakness 1 & Question 1)

  • We supplement the introduction of isolation island literature in the crossover operation in line 240 of the main text. (Reviewer vLRE, Question 2)

  • We supplement the cost accounting scope of the FlowPIE method and conduct cost analysis with other methods in Appendix B.1 (line 694). (Reviewer vLRE, Question 6)

  • We supplement the relationship and differences between LDC and FlowPIE reward feedback in Appendix A (line 668). (Reviewer H999, Question 3)

  • We have revised the titles and descriptions of Table 6 and Table 11 (originally Table 10) to eliminate any potential ambiguity. (Reviewer Xy7E, Weakness 3 & Question 1)


Sincerely thank all reviewers again for your detailed and insightful review. We respectfully look forward to your response.

Best Regards,

Authors of #6495

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.