acceptodds
Under review as a conference paper at ICLR 2027

Ideas Have Genomes: A Comprehensive Benchmark for Scientific Lineage Reasoning and Lineage-Grounded Idea Generation

Abstract

Auto-research systems can retrieve papers, propose hypotheses, and draft research text, but current evaluations rarely ask whether a system understands how scientific ideas descend from prior work. We introduce (Genome-based Evolutionary liNeage Evaluation), a benchmark for evaluating scientific lineage reasoning and lineage-grounded idea generation. The benchmark is built on an explicit representation: a gene is a fine-grained, typed idea atom such as a mechanism, problem niche, limitation, observation, or claim; an is the typed bundle of genes carried by a paper or proposal; and a GenomeDiff aligns genes across predecessor-successor pairs. contains 1,961 golden lineage traces, 1,085 paper genomes, and 920 pairwise GenomeDiffs. It instantiates two complementary evaluations: (42 task types, 1,029 instances) tests closed-form lineage reasoning across genome abstraction, inheritance tracing, evolutionary reasoning, and lineage verification; tests whether generated proposals can be inserted as coherent descendants under Question, Library, and Lineage information settings. Experiments on 15 LLM-based scientists show that current systems struggle with compositional lineage reasoning (GPT-5.5 reaches 23.1% exact accuracy; the best CLI harness reaches 27.3%) and that structured lineage context changes which systems are competitive in generation rather than uniformly improving all systems. reframes evaluation of automated research around what can be claimed from a proposal's relation to prior scientific lineages.

Primary area:
datasets and benchmarks
Keywords:
Idea generation, Auto-research, RSI, Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XamBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission introduces IdeaGene-Bench (IG-Bench), a benchmark for scientific lineage reasoning and lineage-grounded idea generation. It represents papers as minimal, typed, evidence-grounded Idea Genome objects and uses GenomeDiff to record inheritance, mutation, loss, external import, and novel insertion. The IG-Bench supports two types of evaluation: IG-Exam evaluates lineage understanding, while IG-Gym evaluates whether generated proposals inherit the right mechanisms, introduce meaningful variation, and offer selection value. Experimental results show that even the strongest system reaches only 27.3% EM accuracy on the lineage reasoning task, and structured lineage context reshuffles system rankings rather than uniformly helping every participant.

Critical strengths
  1. The submission addresses a meaningful evaluation gap. Whether a proposal/research idea inherits the right mechanism, repairs the right limitation, and stays coherent with the lineage it builds on is an important question beyond retrieval quality, writing fluency, and generic novelty.
  2. Functional role types and evidence pointers provide a useful basis for distinguishing mechanism inheritance from topical proximity.
  3. Combining lineage abstraction, tracing, reasoning, and verification with lineage-grounded proposal generation provides a useful framework for investigating whether plausible research text is supported by coherent reasoning about prior work.
Critical weaknesses
  1. The overall idea is somehow similar to SciPaths: Forecasting Pathways to Scientific Discovery, which already represents scientific progress through enabling contributions, functional roles, evidence-backed rationales, and prior-work grounding. The related-work discussion and positioning table do not account for this closely related paper. The submission should clarify what Idea Genome and GenomeDiff add beyond contribution-level dependency representations, and which evaluation capabilities specifically require these additions.
  2. The instruction to split independently transferable components and retain mutually dependent components leaves substantial room for interpretation. Similarly, LOST needs to distinguish an omitted description from an abandoned component, NOVEL requires an explicit reference scope, and EXTERNAL requires evidence of provenance. These choices directly determine alignments and gold answers, making them central to benchmark validity. They need more clear definitions.
Questions for authors

Please see above.

7 comments
AuthorsOpenReview ↗

Response to Reviewer XamB: Related-Work Positioning, Component Definitions, and Empirical Added Value (Part 1)

Summary

Response focus: related-work positioning, component definitions, and empirical added value.

Thank you for identifying SciPaths as an important related work and for highlighting the need to clarify component boundaries and provenance criteria. We have added comparisons with SciPaths, Intern-Atlas, and ResearchStudio-Idea, and tied component correspondences, changes, and scope to specific evaluation tasks. In the new controls, all six mechanism-replicating proposals receive 18/18 redundancy judgments under each presentation after relevant environment evidence is added. Under a common panel, the Lineage−Library point differences are also positive for all 11 systems across the full set of 30 tasks. Below, we provide the complete controls and operational definitions.

AuthorsOpenReview ↗

Response to Reviewer XamB: Related-Work Positioning, Component Definitions, and Empirical Added Value (Part 2)

W1 (Related work) · Omission of SciPaths and positioning

SciPaths is a closely related and important work: its evidence-supported contribution decomposition provides clear evaluation objects for scientific discovery paths. We have added this work and compare the representation units, relation semantics, and evaluation tasks. Starting from a target contribution, SciPaths predicts the functional components needed to achieve it, then grounds these components in prior papers; its annotations also cover roles, rationales, and cases that cannot be mapped. We share the foundation of decomposing scientific contributions and grounding them in literature evidence.

Our work further makes cross-paper component correspondences and changes themselves evaluation targets. For example, in BERT→RoBERTa, the MLM objective is retained, the masking schedule changes, and NSP is removed from the final recipe; all three can hold within the same paper pair. GenomeDiff separately records the source component, target component, change, and evidence scope for each event, enabling models to trace and verify them individually.

Table X1-1 | Representations and tasks in related work, and our contributions.

WorkRepresentation units and relation semanticsExisting evaluation or applicationQuestions investigated further in our work
SciPathsTarget contributions and functional components, with roles, rationales, and prior-paper sourcesPredict enabling contributions and recover source papersContinue tracing component inheritance, modification, removal, and external introduction from the identified sources, and use these records to evaluate lineage reasoning.
Intern-AtlasNormalized methods and evolutionary relations supported by source passagesReconstruct method lineages; evaluate and generate ideasResolve component correspondences within method-level evolutionary relations, and distinguish driver-mechanism inheritance from niche competition.
ResearchStudio-IdeaInnovation features and transferable ideation patternsLiterature retrieval, prior-work checks, and pattern-guided generationGround ideation strategies in traceable mechanism reuse, limitation–improvement correspondences, and transfer conditions.
Evolving Idea Graphs (EIG)Scientific claims in proposals and their support/conflict relationsLearn graph-editing policies to collaboratively improve proposalsTrace mechanism changes and combinations from multiple sources along cross-paper chains, and evaluate multi-step lineage reasoning.
Graph2IdeaTriples describing problems, methods, mechanisms, and findings in papersGraph-based retrieval context for research planning and ideationAlign scientific objects in retrieved graphs into evidence-supported records of source-to-target component changes.
AgentIdeaBenchLiterature evidence obtained through static reading or active explorationLiterature-verified hypothesis evaluation and thought-experiment studiesEvaluate component understanding, source tracing, and lineage-constrained generation after literature acquisition.
SciMuse (Gu and Krenn)Scientific knowledge graphs, research interests, and personalized ideasInterest assessment and prediction involving more than one hundred research-group leadersExplain proposal value through component provenance, mechanism changes, and the research environment, complementing expert interest assessments.
Idea Genome/GenomeDiff (ours)Typed components with scientific cores, evidence locations, and applicability conditions; cross-paper correspondences record inheritance, modification, removal, external introduction, and additions within a specified scope, including split/merge correspondencesIG-Exam: abstraction, tracing, evolutionary reasoning, and relation verification; IG-Gym: H/V/S and PES within a given research populationEstablish a unified evaluation sequence from component representation through cross-paper reasoning to lineage-constrained generation. Idea Genome specifies scientific functions, evidence locations, and applicability conditions; GenomeDiff records component correspondences, changes, provenance, and comparison scope; driver-mechanism and research-environment criteria support judgments of evolutionary relations. IG-Exam turns these objects into abstraction, tracing, reasoning, and verification questions across 10 domains and 42 task types. IG-Gym reuses the same objects to explain what a proposal inherits, what it changes, and why it has research value within a given research population.
AuthorsOpenReview ↗

Response to Reviewer XamB: Related-Work Positioning, Component Definitions, and Empirical Added Value (Part 3)

These comparisons highlight three specific contributions.

  1. Component changes become direct units of evaluation. GenomeDiff identifies relevant sources and aligns source and target components, recording how their scientific functions persist or change. LOST requires evidence of removal in the target configuration, EXTERNAL requires an import source, and NOVEL specifies an explicit comparison set. These fields directly determine the answers to tracing, reasoning, and verification questions.
  2. Lineage inheritance and the research environment are modeled separately. A shared task places papers in the same environment; evidence of corresponding driver mechanisms establishes inheritance. A proposal can correctly inherit a mechanism while replicating one already present in its environment. We incorporate these two questions separately into relation judgments and proposal evaluation; X2 gives the complete environment-evidence control results.
  3. The same objects connect understanding and generation. IG-Exam measures component abstraction, inheritance tracing, evolutionary reasoning, and relation verification through 1,029 questions spanning 10 domains and 42 task types. IG-Gym further evaluates how generated proposals inherit, change, and adapt to a given research population. Together, they form an evaluation design for scientific lineage capabilities.

These connections make our positioning more precise: Idea Genome/GenomeDiff expresses the development of scientific ideas through locatable components, traceable changes, and assessable relations, and builds evaluations of both understanding and generation around these objects. This positioning is detailed in §2, Table 1, Appendix I, and the references.

W1 (Added value) · Added value and necessity of Idea Genome/GenomeDiff

GenomeDiff expands contribution dependencies into scorable component correspondences, changes, and provenance scope, allowing lineage reasoning to verify, step by step, what is reused, what changes, and whether the same mechanism already exists in the environment. A dependency edge between papers indicates a connection; component records provide the specific information needed to answer the following questions:

Added evaluation objectConcrete questionTasks using this information
Source-to-target component correspondenceWhich operation in the predecessor does the target inherit?IG-Exam T2 tracing; IG-Gym H
Component changes and configuration scopeWhich operations are retained, modified, or removed?IG-Exam T3 reasoning and T4 verification; IG-Gym V
External provenance and research environmentWhere does an addition come from, and does it replicate an existing mechanism?Relation verification; source support and non-redundancy evaluation in IG-Gym

This information lets understanding and generation share the same evaluation objects: Exam tests component tracing and relation verification, while Gym examines inheritance, changes, and research value in proposed work. The new experiments examine the role of this information through research-environment evidence and generation context.

First, relevant environment evidence helps identify mechanism replication while preserving the distinction from substantive changes. We construct 12 proposals from six source groups, hold proposal content fixed, and vary the evidence, yielding 144 judgments. Even if a proposal correctly inherits from its parent paper, it may replicate a mechanism already present in another paper in its environment:

Constructed proposalEvidence providedJudged redundantJudged a distinct mechanismInsufficient evidence
Replicates an existing mechanismPredecessor paper only0153
Replicates an existing mechanismPredecessor paper + irrelevant material0171
Replicates an existing mechanismRelevant environment evidence: natural prose1800
Replicates an existing mechanismSame relevant evidence: JSON1800
Explicitly changes the mechanismPredecessor paper only0171
Explicitly changes the mechanismPredecessor paper + irrelevant material0153
Explicitly changes the mechanismRelevant environment evidence: natural prose0180
Explicitly changes the mechanismSame relevant evidence: JSON0180
Reviewer_HvkLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper introduces IdeaGene-Bench, a benchmark for evaluating whether LLMs can understand how scientific ideas evolve across papers and generate new ideas grounded in existing research. It represents ideas as structured "genomes" and evaluates both lineage reasoning and idea generation.

Critical strengths
  • The problem is interesting and relevant to AI scientists.

  • The benchmark covers multiple scientific domains and evaluates a wide range of LLMs and agent systems.

  • Evaluating whether generated ideas build coherently on prior work is a useful direction.

Critical weaknesses
  • Subjective definitions. The Idea Genome representation and evolutionary categories feel somewhat arbitrary. Scientific ideas are complex, and it is unclear whether these predefined categories can reliably capture their relationships.

  • Bad evaluation. The low accuracy on IG-Exam may partly result from strict exact-match scoring rather than poor scientific reasoning.

  • Presentation. I personally dislike/hate the AI-generated figures. I prefer authors to remake them using hands.

Questions for authors

If my understanding is correct, the paper does not convincingly show that higher lineage-grounded scores correspond to better or more impactful scientific ideas?

4 comments
AuthorsOpenReview ↗

Response to Reviewer HvkL: Scientific Representation, Scoring Granularity, and Content-Based Evaluation (Part 1)

Summary

Response focus: mechanism-level representation, scoring methods, and content validity. Thank you for your suggestions on the complexity of scientific ideas and the scoring approach. The revision explains the representation through component functions and evidence and adds scoring analyses on the same answers: the system-ranking correlation between exact and partial scores is 0.947, and 92.31% of system pairs retain the same order. Agreement over 300 human comparisons is 80%, and the experiment with 84 content-matched pairs also shows no average template bonus. Both main figures were manually drawn in PowerPoint and exported as vector PDFs.

W1 · Subjectivity of the representation and evolutionary categories

Idea Genome starts from the complexity of scientific ideas and makes the fine-grained mechanism relationships intertwined within papers into inspectable objects. A paper can combine multiple sources, retain some mechanisms, modify others, and compete within the same research environment. These relationships require separate descriptions at the component level.

For example, component a1 in paper A and b1 in paper B may exhibit mechanism inheritance, while a2 and b2 may exhibit a substantive modification; B may also import b3 from a third paper. These different component relations jointly form the connections between papers. GenomeDiff records the function, change, provenance, and comparison scope of each correspondence, while the task specifies the evolutionary pattern to be judged.

Components are delineated using scientific function, inputs/outputs, operating conditions, and source evidence. Functions are split into independent components only when the source supports retaining, replacing, or transferring them separately. BERT→RoBERTa is a concrete example: retention of the MLM objective, modification of the masking schedule, and removal of the NSP objective coexist, each with different experimental evidence and scientific meaning.

The definitions also have annotation evidence supporting their reproducible use: pre-adjudication agreement between two PhD students in relevant fields on evolutionary dynamics was 84.7%, with disagreements adjudicated by a third annotator. The revision separately specifies the scope of the label-agreement statistic and the evidence rules for component boundaries. Component granularity depends on the current question and evidence scope, and a single relation label summarizes the specified relation (§3; the Limitations paragraph in §6). Our goal is to follow these locatable component connections and turn the abstract question of how scientific ideas develop into questions that can be traced, verified, and compared.

W2 · Strict whole-question matching and the interpretation of low accuracy

IG-Exam already incorporates multiple measures in its question design and scoring pipeline to reduce the effect of answer format on correctness judgments. Questions include multiple choice, true/false, fixed-label fill-in-the-blank, identifier ordering and grouping, and short matching/mapping tasks. Models can answer with finite choices or short identifiers rather than reproducing a full reference answer. The scorer first extracts the required fields and then normalizes specified labels, abbreviations, and delimiters. Relevant grouping tasks also allow anonymous group names to be exchanged, so these equivalent forms do not incur penalties (§4.2).

We further added partial scores for the same answers to distinguish whether an entire question is correct from how many decisions within it are correct. Exact requires all scored decisions within a question to be jointly correct; Component first computes the proportion of correct components in each question. Scoring granularity changes absolute scores, while the main ordering of systems remains consistent: across all 14 systems, the ranking Spearman ρ is 0.947, 84/91 system pairs retain the same order (92.31%), and the top three remain in the same order. Both scores for each system come from the same set of answers.

Table H2-1 | Exact and partial scores on the same answers. Each system answers 1,029 questions.

SystemExact (%)Partial Component (%)Exact rankPartial-score rank
Claude Code (GPT-5.5)27.2653.9511
Codex (GPT-5.5)24.6252.0722
GPT-5.523.0851.3933
Claude Code (Claude Opus 4.7)21.9849.3946
CoI (GPT-5.5)21.6950.4654
AISv2 (GPT-5.5)21.5750.0165
Qwen3.6-Max-Preview20.5547.4079
Kimi-K2-Thinking20.3847.7388
Gemini-3.1-pro-preview20.1346.16911
Codex (Claude Opus 4.7)19.9148.69107
Claude Opus 4.719.3446.871110
DeepSeek-V4-Pro17.9243.911212
GLM-5.117.6943.701313
MiniMax-M2.711.9334.161414
AuthorsOpenReview ↗

Response to Reviewer HvkL: Scientific Representation, Scoring Granularity, and Content-Based Evaluation (Part 2)

Table H2-2 | Agreement between system rankings under the two scoring methods.

Answer cohortSpearman ρKendall τ_bConcordant system pairsConcordance rate
All 14 systems0.9470.84684/9192.31%

The effect of joint scoring can also be seen in specific task types. Claude Code (GPT-5.5) correctly predicts 80 of the 120 labels in T1-01, giving a component accuracy of 66.67%. Among the 30 questions, all four labels are correct in 5 questions, giving a whole-question accuracy of 16.67%, with all fields successfully parsed. This example distinguishes locally correct judgments from completely solving a question.

Partial scores exceed exact scores by 22.23–28.78 percentage points across all systems. The two metrics respectively ask whether a lineage question is completely solved and how many component judgments are correct, making partial correctness within low whole-question scores explicit (§5.2, Table 3; Appendices F.3–F.4).

W3 · Redrawing the AI-generated figures

We manually drew both Figure 1 and Figure 2 in PowerPoint and exported them as vector PDFs. The new figures replace those in the initial submission, removing the AI-generated decorative icons and little-monster designs. Inspection confirms that neither figure contains raster objects and that all fonts are embedded.

Revised Figure 1 explains the framework through papers, components, and their correspondences. Figure 2 presents evolutionary relations and the Exam/Gym evaluation pipeline. They appear on pages 2 and 6, respectively, with their caption titles and borders marked in blue for ease of review.

Q (scientific quality) · Does higher PES indicate better or more impactful scientific ideas?

PES evaluates the soundness and research value of proposals grounded in prior scientific work; the existing human study supports this proposal-stage evaluation. Across 300 pairwise proposal comparisons within the same task, the model panel agrees with humans' overall preferences in 80% of cases. This relative judgment directly tests which proposal human reviewers prefer.

Specifically, domain experts define H/V/S criteria for each task: H examines inherited mechanisms and their source support; V examines the substance of the proposed changes; and S examines feasibility and research value within the research environment. Genome/Diff provides locatable scientific content and evidence, enabling these judgments to explain what a proposal inherits, what it changes, and why the change is worth investigating. Accordingly, this paper discusses proposal-stage quality. Subsequent experimental success and long-term scientific impact are evaluation targets that require tracking actual research outcomes.

We also examined the relationship between PES and a separate structure-blind LLM quality evaluation. For 216 fixed proposals across six tasks, the quality judges read the task, proposal, and the same source materials, without receiving Genome/Diff, PES, or structural labels:

Scoring methodPairwise ranking concordance (%)
Full PES (with caps)56.76
Raw H/V/S mean61.09

This diagnostic compares the two evaluations in terms of pairwise rankings; 38/216 proposals are affected by caps. Human preference agreement supports correspondence in overall proposal preference, while the structure-blind diagnostic shows ranking changes under different evaluation criteria and capping rules.

Section §4.3 of the revision specifies the evaluation target and aggregation rules; §5.3 and Appendix E.4 report the quality-association diagnostic above; and Appendix D describes the scope of the human preference study. The next response presents content-matched controls to further address whether scoring rewards presentation templates.

AuthorsOpenReview ↗

Response to Reviewer HvkL: Scientific Representation, Scoring Granularity, and Content-Based Evaluation (Part 3)

Q (structural preference) · Preference for benchmark-defined structures and presentation templates

The content-matched paired experiment found no average scoring bonus for the structural template. Each of 84 proposals was rendered as both natural prose and a structural template. Both versions retain all 3,015 scientific-content strings and use the same source and structural evidence. Three judges scored them independently, yielding 504 valid judgments. Mean PES is 80.96 for natural prose and 80.50 for the template.

Table H5-1 | Content-matched comparison of structural templates and natural prose. Scores range from 0–100; differences are “template − natural prose.” Results are equally weighted over 28 tasks or 10 domains, with corresponding cluster resampling.

MetricNatural proseStructural templateTask-weighted difference [95% interval]Domain-weighted difference [95% interval]
PES80.9680.50−0.47 [−0.89, +0.04]−0.49 [−0.85, −0.07]
Raw H/V/S mean81.9581.50−0.46 [−0.87, +0.05]−0.48 [−0.84, −0.06]
H81.4681.18−0.29 [−0.76, +0.27]−0.33 [−0.78, +0.17]
V82.4681.75−0.71 [−1.14, −0.19]−0.71 [−1.08, −0.29]
S80.4880.10−0.38 [−0.84, +0.15]−0.40 [−0.81, +0.01]

Capping records remain identical across all 84 proposal pairs. The average differences under both aggregation schemes do not point to a template reward. The task-level PES interval spans zero, while the domain-level interval is negative.

Two further controls separate content from form. In the relevant-environment-evidence experiment with 12 constructed proposals, presenting the same evidence as natural prose or JSON yields identical categorical judgments in 36/36 proposal–judge pairs. Adding evidence of relevant competing mechanisms, meanwhile, yields 18/18 redundancy judgments for the six mechanism-replicating proposals under each presentation. Removing explicit generation-reasoning blocks while retaining the final scientific proposals for 86 proposals produces 516 valid judgments. The task-macro-average PES changes by less than 0.01 points, with a 95% source-cluster interval of [−0.41, +0.40].

These experiments address scoring-stage preferences for proposal packaging, evidence serialization, and visible reasoning blocks. Sources and already-extracted structures are fixed in the controls; format robustness across the full extraction-to-scoring pipeline is a separate evaluation scope. Full settings and disaggregated results are provided in §5.3–5.4 and Appendices E.6–E.7.

AuthorsOpenReview ↗

Revision · Question formats and scoring analyses, representation definitions, and figures

  • Question formats and scoring (§4.2, §5.2; Table 3; Appendices F.3–F.4): Specify multiple-choice, true/false, fixed-label fill-in-the-blank, and other question types together with answer-normalization rules; add the paired exact/partial-score table and ranking-agreement analysis for 14 systems.
  • Granularity of scientific relations (§3.2–3.5; the Limitations paragraph in §6; Appendix G): Explain how multiple component relations can coexist within a paper pair, and define rules for delineating components using functions, inputs/outputs, and source evidence.
  • Quality-evaluation targets and association diagnostics (§4.3, §5.3; Appendices D and E.4): Describe the proposal-stage H/V/S evaluation targets, the human preference study with 300 pairs, and the structure-blind quality-association diagnostic.
  • Content and form at the scoring stage (§5.3–5.4; Appendices E.6–E.7): Report content-matched template controls, visible-reasoning-block controls, and environment-evidence controls, distinguishing changes in scientific content from changes in presentation.
  • Figures 1 and 2 (pages 2 and 6): Both figures were manually drawn in PowerPoint and exported as vector PDFs, replacing the initial figures and removing the original cartoon designs. Neither contains raster objects, and all fonts are embedded.

Thank you for helping us explain more fully the scientific meaning of the representation, scoring granularity, and the basis of content evaluation.

Reviewer_jMaBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  1. Proposes IdeaGene framework: represents papers as typed Idea Genome objects, aligns predecessor–successor pairs via GenomeDiff recording inheritance/mutation/loss, and defines six evolutionary dynamics.
  2. Builds IG-Bench with IG-Exam (42 task types, 1,029 instances across T1–T4 closed-form reasoning axes) and IG-Gym (10 domains, 30 tasks, 1,260 generated proposals, scored by three LLM judges with H/V/S rubrics → PES under Question / Library / Lineage settings).
  3. Main findings: best system reaches only 27.3% exact accuracy on IG-Exam, with T4 verification hardest; CLI harnesses help retrieval more than compositional consistency; Lineage context separates systems rather than uniformly lifting them.
Critical strengths
  • Mechanism-level lineage is a meaningful gap left open by paper-level QA / novelty evaluation; the framework operationalizes inheritance relations into auditable objects.
  • The benchmark spans both understanding and generation, with varied settings and domains that give it real diagnostic value.
  • Non-trivial data scale with human annotation and multiple judges — not purely synthetic.
Critical weaknesses
  • The T2-04 gold example in Appendix B.1 conflicts with the paper's own definition: neither edge in the NAS chain (DARTS → MnasNet) nor the Inception chain (Inception-v3 → EfficientNet) demonstrates driver-mechanism inheritance, yet both are labeled lineages rather than niche competition. This casts doubt on gold-label validity.
  • Appendix E explicitly states that per-system IG-Gym dimension/domain records could not be reconciled with the Table 2 summaries; the numerical reliability of the reported aggregates is unclear.
  • The three PES judge models are not disclosed; no confidence intervals or repeated-run variability are reported for PES gains or ELO rankings; the 80% human concordance is stated without sample size and concerns pairwise preference, not direct H/V/S validation.
Questions for authors
  • What is the driver-inheritance evidence for each edge in the T2-04 gold answer? Can you report per-task expert accuracy or per-task label agreement?
  • Which IG-Gym source records failed to reconcile, and can you provide corrected tables plus a per-proposal audit trail linking raw judge outputs, caps, and final PES?
  • What are the three PES judge models? What is the sample size and scope of the 80% human concordance, and is there direct human validation of H/V/S/PES?
4 comments
AuthorsOpenReview ↗

Response to Reviewer jMaB: Gold-Label Evidence, Score Traceability, and Evaluation Reliability (Part 1)

Summary

Response focus: label evidence, numerical auditing, and evaluation reliability.

Thank you for your specific comments on gold labels, score traceability, and the scope of validation. We have replaced the appendix examples with examples supported by source passages, reconciled the main table and appendix, and completed score replays for 1,170 historical records and 1,164 records from the new panel, with 0 discrepancies in the score fields. Below, we explain the source evidence, judge configurations, uncertainty, and human validation.

W1 / Q1 · Conflict between the T2-04 gold labels and the definition of driver-mechanism inheritance

We have revised Appendix B.1 according to the definition of driver-mechanism inheritance and reorganized the T2-04 example around individually verifiable component evidence. For the two relationships you identified, DARTS→MnasNet and Inception-v3→EfficientNet, the original illustrative material did not provide sufficient evidence of driver-mechanism inheritance. They are therefore no longer presented as inheritance edges in this example. A shared research direction, chronological order, and common task must be assessed separately from evidence that a specific mechanism is inherited.

The revised example uses BERT→RoBERTa, which clearly illustrates three distinct component events within the same paper pair:

ComponentEvidence in the source paperEvidence in the target paperChange within the specified scope
MLM prediction objectiveBERT §3.1 defines the masked language modeling objectiveRoBERTa retains this objective and studies its training recipeInheritance of the objective's function
Masking scheduleBERT uses preconstructed masking in its training examplesRoBERTa §4.1 compares static and dynamic maskingModification of the masking schedule
NSP auxiliary objectiveBERT §3.1 defines NSPRoBERTa §5 explicitly removes NSP from the final training recipeLOST in the final recipe

The gold labels were annotated and reviewed by two PhD students in relevant fields, with judgments grounded in the components, configurations, and source passages specified by each question. Scientific papers contain intertwined component histories: the same paper pair can involve objective inheritance, mechanism modification, component removal, and competition within a research environment. Idea Genome represents these connections separately, while each closed-form question specifies the mechanism and scope to be judged. This is the rationale for a component-level representation.

Sections §3.4–3.5 of the revision clarify the criteria for inheritance and evolutionary relations. Appendix B.1 provides the four updated types of illustrative questions and their source evidence; Appendix G.1 specifies component and configuration scope; and the Limitations paragraph in §6 discusses the simplification of multiple relations. The appendix examples have been replaced. The completed expert-review decisions will be incorporated into the question bank and corresponding scores through item-level version records. The currently reported scores use the released question and gold-label versions.

Q1 · Per-task expert accuracy and label agreement

To address label reliability, we report an overall pre-adjudication agreement of 84.7% for evolutionary-dynamics annotations and specify the annotation, review, and adjudication process. Two PhD students in relevant fields annotated and reviewed the gold labels against the original papers, with disagreements adjudicated by a third annotator.

This statistic measures agreement between annotators on evolutionary-dynamics labels before adjudication, directly addressing whether the categories have reproducible decision criteria. Its scope is overall label annotation across tasks. Expert answering accuracy instead measures experts' performance on finalized questions; these concern different stages of the study. Section §4.1 and Appendix D of the revision explicitly preserve this measured object and level of aggregation.

Beyond reporting agreement, the revision provides verifiable annotation criteria: Appendix B.1 presents component events and source locations, while Appendix G.1 specifies component splitting, removal evidence for LOST, the comparison scope for NOVEL, and import evidence for EXTERNAL. Overall agreement and concrete evidence rules jointly explain how the gold labels were established. The human preference comparisons involving 300 proposal pairs are discussed separately in the human-validation response below.

AuthorsOpenReview ↗

Response to Reviewer jMaB: Gold-Label Evidence, Score Traceability, and Evaluation Reliability (Part 2)

W2 / Q2 · Appendix E, Table 2, and the per-proposal audit trail

Per-proposal replay has reproduced the historical main table and confirmed consistent score computation for the new common panel. Each proposal can now be traced to individual judges' scores and rationales, followed by H/V/S, dimension-level caps, the proposal-level cap, and final PES.

Scoring cohortRecords coveredItems checkedResult
Historical main table13 systems, 1,170 proposal records8 intermediate/final score fields and system aggregates0 discrepancies in score fields; reproduced the displayed main-table values
New common panel1,164 proposals, 3,492 valid judgmentsScoring requests, raw responses, parsed results, aggregation, and caps0 discrepancies in score fields

The Codex score of 86.7 comes from a separate batch verified by the author team. Its Question, Library, and Lineage means are 84.123, 87.456, and 88.521, respectively, with an equally weighted average of 86.700. This result remains in the main table and is identified separately from the other local batches.

The difference between dimension aggregates and final PES can be checked along the calculation trail. For example, a Gemini / Spatial Transcriptomics / Question proposal has raw H/V/S scores of 78.22 / 82.58 / 81.25 and a raw PES of 80.69. The proposal claims to inherit 2 components, but automated source matching finds 0 matches in the inferred source, Nicheformer, triggering the predefined inheritance rule. H is first capped at 35, reducing the dimension mean to (35 + 82.58 + 81.25) / 3 = 66.28. The PES cap is then applied, giving min(66.28, 50) = 50.00. The cap of 50 is specified in advance by the scoring protocol. Here, the trigger comes from automated structural matching; the audit record retains the raw judge scores, matching result, applicable caps, and final score.

The additional judge scores have been recomputed using the common panel of GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro Preview. Of the 1,170 historical proposals, 1,164 were rescored by the new panel without changing their content, yielding 2,334 scoring records across batches for 1,170 distinct proposals. The historical main table and new panel are aggregated separately. The per-proposal audit materials retain the correspondence, calculation order, and rounding rules (Table 2; Appendices C.1 and E).

W3 / Q3 · PES judge identities and evaluation protocol

The PES judges in the current common-panel rescoring are GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro Preview. Each judge scores the same proposal separately, using the following request configurations across all evaluated systems:

Judge modelTemperatureOutput limit (tokens)
GPT-5.514,096
Claude Opus 4.704,096
Gemini 3.1 Pro Preview04,096

The common panel has completed 1,164 proposals and 3,492 valid judgments. The Lineage−Library comparison in this response covers all 30 tasks, 11 systems with complete records, and three settings, totaling 990 proposals.

Judges read the task rubric, proposal, and fixed source and structural evidence, then provide H/V/S subcriterion scores and rationales. Valid scores are aggregated by subcriterion, followed by dimension scores, dimension caps, the mean of the three dimensions, and the proposal cap to obtain final PES. The configurations, coverage, and aggregation order are specified in §4.3 and Appendices C.1–C.2 of the revision. The correspondence between scoring records is explained in the preceding audit response.

AuthorsOpenReview ↗

Response to Reviewer jMaB: Gold-Label Evidence, Score Traceability, and Evaluation Reliability (Part 3)

W3 · PES/ELO intervals, rankings, and repeated-run variation

We have added paired intervals, repeated scoring, repeated generation, and ELO resampling results so that point estimates and variation can be assessed together. For example, under the common panel, GPT-5.5 has a Lineage−Library PES difference of +0.905 points, with a 95% source-cluster interval of [0.381, 1.505]. The table below reports both the context gain and a comparison between leading systems:

ComparisonAll 30 tasks (11 systems): PES difference [95% interval]
GPT-5.5: Lineage − Library+0.905 [0.381, 1.505]
GPT-5.5 − Claude Code (GPT-5.5)+0.097 [−0.831, 1.392]

The intervals come from 10,000 paired source-cluster resamples of fixed proposals and are reported pointwise. The leading systems' intervals overlap, and their point-estimate ranking reverses when the Gemini judge is removed; the intervals therefore inform the assessment of their relative lead.

Repetition studyComplete coverageRepetitions per unitMean within-unit sample SD of PESMedian SD [25th–75th percentiles]
Fixed proposals, repeated scoring84 proposals, 28 tasks; 756 judge decisions in total3 scoring rounds0.6710.560 [0.349, 0.900]

The study in the table quantifies scoring variation for fixed proposals. All categorical decisions in this cohort are Yes/Accept, and the brackets give the 25th and 75th percentiles of the per-proposal standard deviations. Variation in the repeated-generation study includes both generation differences and judge noise; its full settings and distribution are provided in Appendix E.3.

For ELO, we recovered 910 historical Lineage matches across 10 tasks. A model with 14 nodes, including PaperQA, reproduces the archived scores for the 13 systems. Using 2,000 refits over 8 shared-source clusters, GPT-5.5 has an ELO of 1213.2, with a conditional 95% interval of [1145.2, 1284.4]. The historical rule counted 324 invalid judgments among 2,730 judge positions as ties; the resampling retains and discloses this rule. The complete protocol is provided in §5.3 and Appendices C.3 and E.2–E.3.

W3 / Q3 · The 80% human agreement and direct H/V/S validation

The 80% agreement is based on 300 pairwise proposal comparisons and measures agreement between humans and the model panel on overall preference. Human reviewers compare two proposals for the same task. Such relative judgments facilitate evaluation at scale and reduce the difficulty of applying a common absolute scoring scale across tasks.

The three components serve different purposes: domain experts define task-specific H/V/S criteria; the 300 pairwise comparisons provide evidence of overall preference agreement; and Krippendorff's α=0.74 measures agreement among PES judges. The current human-validation conclusions concern overall proposal preference. The study scope and statistical definitions are specified in §4.1 and Appendix D.

AuthorsOpenReview ↗

Revision · Gold-label examples, score traceability, and reliability

  • Component inheritance evidence (§3.4–3.5; Appendices B.1 and G.1): Replace the original illustrative question with BERT→RoBERTa, providing source evidence for MLM inheritance, masking modification, and NSP removal. The Limitations paragraph in §6 explains multiple component relations within a paper pair. Section §4.1 specifies the question and gold-label versions used for the reported scores.
  • Gold labels and human studies (§4.1; Appendix D): Describe annotation and review by two PhD students in relevant fields, disagreement adjudication by a third annotator, and the overall pre-adjudication label agreement of 84.7%. The 80% preference agreement over 300 pairs and the PES inter-judge α=0.74 are identified by their respective measured objects.
  • Scoring protocol and audit (§4.3, §5.1; Table 2; Appendices C.1–C.2 and E): Disclose the three common-panel judges and their configurations, explain the correspondence between main-table and common-panel rescoring records, and provide the calculation trail from raw scores to capped PES.
  • Intervals and repetition (§5.3; Appendices C.3 and E.2–E.3): Add PES/ELO intervals, judge sensitivity, and both repetition studies, separately presenting variation from scoring fixed content and regenerating proposals.

Thank you for encouraging a clearer presentation of label evidence, numerical calculation trails, and evaluation scope.

Comments on the paper

1 comment
AuthorsOpenReview ↗

IG-Bench General Contributions and Revisions

Dear Area Chair and Reviewers jMaB, HvkL, and XamB,

Thank you for pressing us on what counts as scientific inheritance and how reliably we can measure it. IG-Bench connects two questions. Can a system trace a specific mechanism from one paper to the next? Can it use that history to propose a change that addresses a limitation of prior work?

What the representation adds. Idea Genome identifies components by their scientific function and supporting passages. GenomeDiff follows those components across papers. For example, RoBERTa retains BERT’s masked-language-modeling objective, changes the masking schedule, and removes the next-sentence-prediction objective from its final training recipe. Each claim points to different evidence, even though all three concern the same paper pair. In response to jMaB and XamB, we replaced the disputed appendix example, specified the evidence required for each type of change, and clarified how this extends the contribution-level dependencies studied by SciPaths. The reported Exam scores use the released questions and labels.

What systems can trace. IG-Exam tests lineage reasoning on 1,029 questions across 42 task types. To address HvkL’s concern about exact matching, we scored the same answers at two levels. The strongest system answered 27.26% of questions completely correctly and recovered 53.95% of the scored components on average. Partial credit changes the interpretation of the absolute scores, while 84 of 91 pairwise system rankings are preserved (Table 3).

What lineage contributes to generation. IG-Gym asks whether a proposal inherits a supported mechanism, makes a substantive change, and remains viable in its research setting. Across all 30 tasks, all 11 systems with complete records had higher mean proposal scores with lineage context than with the paper library alone. Pointwise 95% intervals excluded zero for eight systems. To make these results auditable, we report the judge configurations and scoring rules, trace the historical main-table means for 13 systems to 1,170 proposal records, and report the common-panel rescoring separately. Across 300 proposal pairs, the panel agreed with human reviewers’ overall preference in 80% of comparisons (Appendices C–E).

We appreciate your feedback. It helped us state more precisely which scientific relationships the benchmark captures and what its scores tell us about current systems. If the revised paper and additional analyses address your concerns, we would be grateful if you would consider this progress when revisiting your ratings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.