acceptodds
Under review as a conference paper at ICLR 2027

Forming Feasible Research Ideas by Analogical Transfer over Rationale-Bearing Graphs

Abstract

Large language models are increasingly used to generate research ideas, yet human studies find their ideas weakest on feasibility. Existing work scores or tests feasibility once an idea exists; how the idea takes shape has received little attention. We present IdeaStrata, a harness that addresses feasibility while the idea is formed. Cognitive theories of analogy ask of a sound transfer whether a relation, with the conditions under which it holds, remains valid in a new setting, the question on which feasibility turns; IdeaStrata makes this check the representation the model operates on and the steps it carries out. Each input paper becomes a Hierarchical Layered Graph whose edges carry a rationale, the mechanism and conditions under which a relation holds; rationale transfer maps a relation onto concepts of another paper only when those conditions still hold, and structural verification removes transferred relations that cannot hold together. An idea is written only from relations that pass both checks. On IdeaBench with two backbones, IdeaStrata generates the most feasible ideas, with a mean feasibility 1.8x (GPT-4o) and 2.4x (Qwen3.8-Flash) that of the strongest baseline, and shifts the whole feasibility distribution upward, including target papers for which all three generated ideas are judged more feasible than the target itself. Its ideas are least often criticized for the concerns the design targets, such as a relation applied where its conditions do not hold, and removing either the rationale or structural verification lowers feasibility and raises these concerns.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
scientific idea generation, research ideation, feasibility, analogical reasoning, structure-mapping theory, agent harness, large language models, structured representation, AI for science

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CszcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose using analogy to propose research ideas, based on the assumption that analogical transfer will identify more feasible hypotheses. The authors define the problem as receiving the top cited papers for a target paper, then building a relational graph from the references and uses analogical induction to predict edges (relationships) for the hypothesis. The authors validate their work on IdeaBench using two models against multiple baselines, and include an ablation experiment of removing analogical steps.

Critical strengths

The experiments include ablations on the main method and multiple baselines

Critical weaknesses

The results seem significant given the size of the corpus, but the improvements are measured using only one judge model in a single trial, so it is hard to tell how much specific model selection influences the validity of these results. The method of graph extraction and prediction through analogy need to be explored further. It is unclear how this approach compares to other edge prediction methods, or how sensitive it is to the specific domain. The method does not limit itself to IdeaBench since it can always select top five citations from any other paper, so a breakdown by research field could be informative.

Questions for authors

What other test data could you include in addition to IdeaBench? How did you evaluate the relationship extraction method, how does this perform across research fields?

Reviewer_zHKzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IdeaStrata, a framework for generating feasible research ideas through cross-paper analogical transfer. It represents papers as hierarchical graphs whose edges encode mechanisms and applicability conditions, then constructs ideas through transfer checks and structural consistency verification. On IdeaBench with two backbone models, IdeaStrata outperforms the evaluated baselines on LLM-judged feasibility. Ablation studies show that both edge rationales and structural verification contribute to these gains.

Critical strengths

[S1] The paper addresses feasibility, an important challenge in research idea generation. Its focus on checking mechanisms and applicability conditions during idea formation provides a well-motivated approach that can complement subsequent evaluation or execution.

[S2] The representation of mechanisms and conditions as edge rationales connects naturally to the proposed transfer checks. Separating the admissibility of individual transfers from the consistency of their combination gives the framework a clear structure grounded in theories of analogy.

[S3] On IdeaBench with 2,374 target papers, IdeaStrata improves LLM-judged feasibility over the evaluated baselines with both backbones. The concern analysis and ablations provide additional evidence that edge rationales and structural verification contribute to the reported gains.

Critical weaknesses

[W1] Although Section 3.2 defines three abstraction levels, it provides limited operational guidance for selecting node granularity and distinguishing composable concepts from operational components. The paper also lacks an assessment of extraction quality or sensitivity to granularity, leaving the reliability of the representation underlying subsequent stages unclear.

[W2] Sections 3.2–3.4 provide an illustrative example of rationale transfer, but do not identify it as an actual experimental output or connect it to specific input papers. An end-to-end case showing extracted relations, transfer decisions, verification outcomes, and the final idea would help establish how the proposed mechanism operates in practice.

[W3] Generation experiments use only GPT-4o and Qwen3.8-Flash, with the latter configured at low reasoning effort; GPT-5.6-sol is used only for evaluation. It remains unclear whether the improvements persist with stronger generation models or higher reasoning budgets. Such comparisons would help establish the broader applicability of the framework.

[W4] The paper reports no independent evaluation of the generated ideas by domain experts. Both feasibility rankings and concern tagging rely on GPT-5.6-sol, with the latter derived from the same judge’s comments. Given the central claim of improved feasibility, a blinded expert evaluation on a representative subset is needed to assess whether the reported gains align with human judgments of scientific and practical feasibility. I would be willing to raise my rating to 3 if the authors conduct such an evaluation and provide an in-depth analysis of the findings.

Questions for authors

How reliable is LLM-based reasoning over the extracted graphs? Could the authors directly evaluate the correctness of transferred relations and the accuracy of inconsistency detection, beyond downstream idea-level feasibility scores? Have they considered comparisons with graph-mining methods for proposing candidate relations or symbolic constraint checking for verification? Such comparisons would help clarify the strengths and limitations of relying on LLMs for these graph reasoning operations.

Reviewer_wqnKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces IdeaStrata, an framework designed to enhance the feasibility of scientific research ideas generated by LLMs through structured reasoning. Not like traditional approaches, IdeaStrata integrates feasibility verification directly into the idea-generation process. Specifically, the framework represents scientific papers as HLGs, where nodes encompass scientific concepts at varying levels of abstraction, and edges encode the relationships between these concepts, alsop including the underlying mechanisms and the conditions under which these relationships hold. The framework consists of three stages: reasoning skill extraction, logical rationale transfer, and structured verification. Through these stages, the model can not only transfer relationships across different papers but also simultaneously verify the robustness of underlying assumptions and ensure internal consistency among the derived relationships. Experiments conducted on IdeaBench, both using GPT-4o and Qwen3.8-Flash demonstrating that IdeaStrata outperforms baselines in improving the feasibility of LLM-generated research ideas.

Critical strengths

Experiments conducted on IdeaBench, both using GPT-4o and Qwen3.8-Flash demonstrating that IdeaStrata outperforms baselines in improving the feasibility of LLM-generated research ideas.

Critical weaknesses

I believe the paper has several notable shortcomings:

  1. The core claim of this work is that IdeaStrata can generate more feasible scientific research ideas. However, its evaluation relies entirely on a LLM acting as a "judge" to rank the generated ideas against existing target papers, rather than verifying whether these ideas are truly actionable or if their underlying assumptions hold up in practice. Scientific feasibility depends on a variety of factors—such as computational resources, experimental constraints, data availability, and implementation complexity—which are often difficult to assess reliably based solely on textual descriptions.
  2. There is a lack of evaluation regarding the trade-off between "feasibility" and "novelty." While enhancing feasibility is a primary objective, a valuable scientific research idea must also possess a certain degree of novelty and potential scientific impact.
  3. IdeaStrata introduces several potentially time-intensive processing stages, including paper-level graph construction, argument extraction, candidate relationship mapping, condition verification, and global consistency checks. Given that the framework must process multiple scientific documents and a potentially vast number of candidate relationships, its computational cost is likely to increase significantly as the number and complexity of the input papers grow. Yet, the paper provides no detailed comparative analysis against baseline methods regarding token consumption, inference latency, or overall computational cost.
Questions for authors

No

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.