- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This submission introduces IdeaGene-Bench (IG-Bench), a benchmark for scientific lineage reasoning and lineage-grounded idea generation. It represents papers as minimal, typed, evidence-grounded Idea Genome objects and uses GenomeDiff to record inheritance, mutation, loss, external import, and novel insertion. The IG-Bench supports two types of evaluation: IG-Exam evaluates lineage understanding, while IG-Gym evaluates whether generated proposals inherit the right mechanisms, introduce meaningful variation, and offer selection value. Experimental results show that even the strongest system reaches only 27.3% EM accuracy on the lineage reasoning task, and structured lineage context reshuffles system rankings rather than uniformly helping every participant.
Critical strengths
- The submission addresses a meaningful evaluation gap. Whether a proposal/research idea inherits the right mechanism, repairs the right limitation, and stays coherent with the lineage it builds on is an important question beyond retrieval quality, writing fluency, and generic novelty.
- Functional role types and evidence pointers provide a useful basis for distinguishing mechanism inheritance from topical proximity.
- Combining lineage abstraction, tracing, reasoning, and verification with lineage-grounded proposal generation provides a useful framework for investigating whether plausible research text is supported by coherent reasoning about prior work.
Critical weaknesses
- The overall idea is somehow similar to SciPaths: Forecasting Pathways to Scientific Discovery, which already represents scientific progress through enabling contributions, functional roles, evidence-backed rationales, and prior-work grounding. The related-work discussion and positioning table do not account for this closely related paper. The submission should clarify what Idea Genome and GenomeDiff add beyond contribution-level dependency representations, and which evaluation capabilities specifically require these additions.
- The instruction to split independently transferable components and retain mutually dependent components leaves substantial room for interpretation. Similarly, LOST needs to distinguish an omitted description from an abandoned component, NOVEL requires an explicit reference scope, and EXTERNAL requires evidence of provenance. These choices directly determine alignments and gold answers, making them central to benchmark validity. They need more clear definitions.
Questions for authors
Please see above.
7 comments
Response to Reviewer XamB: Related-Work Positioning, Component Definitions, and Empirical Added Value (Part 2)
W1 (Related work) · Omission of SciPaths and positioning
SciPaths is a closely related and important work: its evidence-supported contribution decomposition provides clear evaluation objects for scientific discovery paths. We have added this work and compare the representation units, relation semantics, and evaluation tasks. Starting from a target contribution, SciPaths predicts the functional components needed to achieve it, then grounds these components in prior papers; its annotations also cover roles, rationales, and cases that cannot be mapped. We share the foundation of decomposing scientific contributions and grounding them in literature evidence.
Our work further makes cross-paper component correspondences and changes themselves evaluation targets. For example, in BERT→RoBERTa, the MLM objective is retained, the masking schedule changes, and NSP is removed from the final recipe; all three can hold within the same paper pair. GenomeDiff separately records the source component, target component, change, and evidence scope for each event, enabling models to trace and verify them individually.
Table X1-1 | Representations and tasks in related work, and our contributions.
| Work | Representation units and relation semantics | Existing evaluation or application | Questions investigated further in our work |
|---|---|---|---|
| SciPaths | Target contributions and functional components, with roles, rationales, and prior-paper sources | Predict enabling contributions and recover source papers | Continue tracing component inheritance, modification, removal, and external introduction from the identified sources, and use these records to evaluate lineage reasoning. |
| Intern-Atlas | Normalized methods and evolutionary relations supported by source passages | Reconstruct method lineages; evaluate and generate ideas | Resolve component correspondences within method-level evolutionary relations, and distinguish driver-mechanism inheritance from niche competition. |
| ResearchStudio-Idea | Innovation features and transferable ideation patterns | Literature retrieval, prior-work checks, and pattern-guided generation | Ground ideation strategies in traceable mechanism reuse, limitation–improvement correspondences, and transfer conditions. |
| Evolving Idea Graphs (EIG) | Scientific claims in proposals and their support/conflict relations | Learn graph-editing policies to collaboratively improve proposals | Trace mechanism changes and combinations from multiple sources along cross-paper chains, and evaluate multi-step lineage reasoning. |
| Graph2Idea | Triples describing problems, methods, mechanisms, and findings in papers | Graph-based retrieval context for research planning and ideation | Align scientific objects in retrieved graphs into evidence-supported records of source-to-target component changes. |
| AgentIdeaBench | Literature evidence obtained through static reading or active exploration | Literature-verified hypothesis evaluation and thought-experiment studies | Evaluate component understanding, source tracing, and lineage-constrained generation after literature acquisition. |
| SciMuse (Gu and Krenn) | Scientific knowledge graphs, research interests, and personalized ideas | Interest assessment and prediction involving more than one hundred research-group leaders | Explain proposal value through component provenance, mechanism changes, and the research environment, complementing expert interest assessments. |
| Idea Genome/GenomeDiff (ours) | Typed components with scientific cores, evidence locations, and applicability conditions; cross-paper correspondences record inheritance, modification, removal, external introduction, and additions within a specified scope, including split/merge correspondences | IG-Exam: abstraction, tracing, evolutionary reasoning, and relation verification; IG-Gym: H/V/S and PES within a given research population | Establish a unified evaluation sequence from component representation through cross-paper reasoning to lineage-constrained generation. Idea Genome specifies scientific functions, evidence locations, and applicability conditions; GenomeDiff records component correspondences, changes, provenance, and comparison scope; driver-mechanism and research-environment criteria support judgments of evolutionary relations. IG-Exam turns these objects into abstraction, tracing, reasoning, and verification questions across 10 domains and 42 task types. IG-Gym reuses the same objects to explain what a proposal inherits, what it changes, and why it has research value within a given research population. |
Response to Reviewer XamB: Related-Work Positioning, Component Definitions, and Empirical Added Value (Part 3)
These comparisons highlight three specific contributions.
- Component changes become direct units of evaluation. GenomeDiff identifies relevant sources and aligns source and target components, recording how their scientific functions persist or change. LOST requires evidence of removal in the target configuration, EXTERNAL requires an import source, and NOVEL specifies an explicit comparison set. These fields directly determine the answers to tracing, reasoning, and verification questions.
- Lineage inheritance and the research environment are modeled separately. A shared task places papers in the same environment; evidence of corresponding driver mechanisms establishes inheritance. A proposal can correctly inherit a mechanism while replicating one already present in its environment. We incorporate these two questions separately into relation judgments and proposal evaluation; X2 gives the complete environment-evidence control results.
- The same objects connect understanding and generation. IG-Exam measures component abstraction, inheritance tracing, evolutionary reasoning, and relation verification through 1,029 questions spanning 10 domains and 42 task types. IG-Gym further evaluates how generated proposals inherit, change, and adapt to a given research population. Together, they form an evaluation design for scientific lineage capabilities.
These connections make our positioning more precise: Idea Genome/GenomeDiff expresses the development of scientific ideas through locatable components, traceable changes, and assessable relations, and builds evaluations of both understanding and generation around these objects. This positioning is detailed in §2, Table 1, Appendix I, and the references.
W1 (Added value) · Added value and necessity of Idea Genome/GenomeDiff
GenomeDiff expands contribution dependencies into scorable component correspondences, changes, and provenance scope, allowing lineage reasoning to verify, step by step, what is reused, what changes, and whether the same mechanism already exists in the environment. A dependency edge between papers indicates a connection; component records provide the specific information needed to answer the following questions:
| Added evaluation object | Concrete question | Tasks using this information |
|---|---|---|
| Source-to-target component correspondence | Which operation in the predecessor does the target inherit? | IG-Exam T2 tracing; IG-Gym H |
| Component changes and configuration scope | Which operations are retained, modified, or removed? | IG-Exam T3 reasoning and T4 verification; IG-Gym V |
| External provenance and research environment | Where does an addition come from, and does it replicate an existing mechanism? | Relation verification; source support and non-redundancy evaluation in IG-Gym |
This information lets understanding and generation share the same evaluation objects: Exam tests component tracing and relation verification, while Gym examines inheritance, changes, and research value in proposed work. The new experiments examine the role of this information through research-environment evidence and generation context.
First, relevant environment evidence helps identify mechanism replication while preserving the distinction from substantive changes. We construct 12 proposals from six source groups, hold proposal content fixed, and vary the evidence, yielding 144 judgments. Even if a proposal correctly inherits from its parent paper, it may replicate a mechanism already present in another paper in its environment:
| Constructed proposal | Evidence provided | Judged redundant | Judged a distinct mechanism | Insufficient evidence |
|---|---|---|---|---|
| Replicates an existing mechanism | Predecessor paper only | 0 | 15 | 3 |
| Replicates an existing mechanism | Predecessor paper + irrelevant material | 0 | 17 | 1 |
| Replicates an existing mechanism | Relevant environment evidence: natural prose | 18 | 0 | 0 |
| Replicates an existing mechanism | Same relevant evidence: JSON | 18 | 0 | 0 |
| Explicitly changes the mechanism | Predecessor paper only | 0 | 17 | 1 |
| Explicitly changes the mechanism | Predecessor paper + irrelevant material | 0 | 15 | 3 |
| Explicitly changes the mechanism | Relevant environment evidence: natural prose | 0 | 18 | 0 |
| Explicitly changes the mechanism | Same relevant evidence: JSON | 0 | 18 | 0 |
Response to Reviewer XamB: Related-Work Positioning, Component Definitions, and Empirical Added Value (Part 1)
Summary
Response focus: related-work positioning, component definitions, and empirical added value.
Thank you for identifying SciPaths as an important related work and for highlighting the need to clarify component boundaries and provenance criteria. We have added comparisons with SciPaths, Intern-Atlas, and ResearchStudio-Idea, and tied component correspondences, changes, and scope to specific evaluation tasks. In the new controls, all six mechanism-replicating proposals receive 18/18 redundancy judgments under each presentation after relevant environment evidence is added. Under a common panel, the Lineage−Library point differences are also positive for all 11 systems across the full set of 30 tasks. Below, we provide the complete controls and operational definitions.