acceptodds
Under review as a conference paper at ICLR 2027

Evolving Idea Graphs with Learnable Edits-and-Commits for Multi-Agent Scientific Ideation

Abstract

LLM-empowered multi-agent systems offer new potential to accelerate scientific discovery by generating novel research ideas. However, existing methods typically coordinate agents through temporary texts, such as drafts or chat logs; it is difficult to pinpoint the weaknesses in the generated ideas and how the agents refine them. To this end, we introduce Evolving Idea Graphs (EIG), a graph-based multi-agent scientific ideation framework that can generate high-performance research ideas across various benchmark-native metrics, such as novelty, feasibility, and clarity. Instead of coordinating solely through texts, EIG represents a partially formed proposal as an evolving idea graph, where nodes capture scientific claims and edges encode relations (e.g., support and conflict), enabling unresolved weaknesses to remain identifiable throughout the idea evolving process. Specifically, a learned two-head controller operates over the evolving graph to guide the ideation: one head selects graph edits for agents to execute, while the other decides when the graph is ready for commit as final proposal synthesis. On AI Idea Bench 2025 and LiveIdeaBench, EIG outperforms all compared systems on both automatic benchmark scores and blinded expert ratings. Ablations show that the graph-based state, its typed relations, and the learned edit-and-commit control each contribute to final proposal quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
idea generation, multi-agent system

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dicUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Multi-agent scientific ideation often hides unsupported claims and contradictions in text-based collaboration. EIG maintains a typed idea graph and learns a two-head controller for parallel graph edits and stopping. On 512 held-out tasks from AI Idea Bench 2025 and LiveIdeaBench, it reports higher automatic scores than the compared baselines and better overall expert ratings on 24 tasks.

Critical strengths
  1. The problem is important and well motivated. Localizing flaws and tracking improvements while building research proposals is a valuable goal. Typed scientific claims and relations in the idea graph maintain visible weaknesses throughout idea evolution to support agent collaboration.

  2. The paper is generally clearly written and easy to understand.

  3. The evaluation is comprehensive. It includes several ablations, portability checks with two additional generators, and blinded expert evaluation independent of the automatic scorers.

Critical weaknesses
  1. Why does the learned controller outperform its heuristic teacher that provides its training labels? The controller is trained to imitate the heuristic teacher, but Table 4 reports average scores of 7.28 without calibration and 7.41 for full EIG, both exceeding the 6.89 achieved by direct heuristic control. Although Appendix E.9 reports that the full learned policy changes 37.7% of edit choices, it does not establish which changes produce better outcomes or how the retained calibration, fallbacks, and safeguards contribute.

  2. The semantic validation of graph relations remains unclear. Appendix K provides a useful refinement example and supporting literature, but does not clearly connect that literature to the evidence attached during execution or illustrate how an explicit contradiction is resolved. Could the authors clarify these connections using concrete node content and edits, and explain how relation validity and contradiction resolution are assessed?

  3. More implementation details would improve reproducibility. Please provide more details such as the rules for generating teacher stopping labels, deployment stopping thresholds, calibration bounds, and how the training losses are combined. It would also be helpful to clarify which fallback rules and safeguards are retained in NoCalibration.

Questions for authors

See weaknesses

4 comments

Response to Reviewer dicU (1/3)

[W1 Why the learned controller outperforms its heuristic teacher, and how calibration, fallback, and safeguards contribute]

The critic is trained on the teacher's within-slate preferences and learns beyond them. The teacher labels the best candidate in each slate from a one-step simulation on four graph signals, while the critic scores candidates from the full graph, the target node's neighbourhood, and the candidate and state text. The trained critic agrees with the teacher's choice on 60.7% of held-out edit decisions, and the final policy departs from the heuristic on 37.7% of edits on identical states. EIG beats the heuristic by +0.39 pooled on identical states and +0.52 end to end, and calibration adds a further +0.13.

  1. Gap to the teacher, two ways. On identical logged states and candidate slates, the two policies differ only in the decision rule (Table R10, response to Reviewer 75H6). End to end, each policy runs its own trajectory (Table 4). Intervals are 95% paired bootstrap intervals over groups; the learned policy's quality passes the teacher's with 100 of the 400 training groups (Table R8, response to Reviewer mTBr).

    Table R17 — EIG minus the heuristic teacher (512 groups, 3 seeds)

    ComparisonAIIBLivePooled
    Identical states+0.45 [0.38, 0.52]+0.34 [0.26, 0.42]+0.39 [0.34, 0.44]
    End to end+0.64 [0.57, 0.71]+0.40 [0.32, 0.47]+0.52 [0.47, 0.57]
  2. Where the gain is concentrated. For each of the 512 groups, we take the share of its audited edit decisions on which EIG differs from the heuristic, and the group's matched-state score gap. Groups with at least one differing edit are split into thirds by that share.

    Table R18 — Matched-state gap by how much EIG departs from the heuristic

    Groups, by share of differing editsGroupsAudited editsDiffering editsGap, EIG − heuristic [95% CI]
    No differing edit436000+0.01 [−0.01, 0.03]
    Lowest third of the rest (mean share 26%)1561,959521+0.09 [0.04, 0.14]
    Middle third (39%)1562,114818+0.31 [0.24, 0.39]
    Highest third (58%)1572,0951,213+0.88 [0.77, 0.99]

    Groups in which no audited edit differs have a gap near zero (+0.01), and the gap grows with the share of edits on which the two policies differ.

  3. Contribution of each component. Replacing learned edit control by the heuristic gives 6.94 (−0.47 against EIG), and replacing learned commit control by a fixed horizon gives 7.09 (−0.32). The ladder below adds the runtime mechanisms; all rows are on the 512 groups.

    Table R19 — From the teacher rule to the full system

    VariantDescriptionAvg.Δ vs EIG-Heuristic
    EIG-Heuristicteacher rule6.89—
    Unrestricted criticlearned critic; phase mask, fallback, calibration, and safeguards removed7.01+0.12
    EIG-NoCalibrationlearned critic with mask, fallback, and safeguards7.28+0.39
    EIG (Ours)adds calibration7.41+0.52

    Calibration contributes +0.13 (+0.11 AIIB, +0.14 Live). The learned critic scores above the teacher (7.01 against 6.89) even with the phase mask, fallback, calibration, and safeguards removed, and adding the mask, fallback, and safeguards brings it to 7.28.

Response to Reviewer dicU (2/3)

[W2 Semantic validation of graph relations, and how a contradiction is resolved]

Relations are validated against the idea's semantic structure when they are proposed. A candidate must connect existing idea nodes, use an action and a relation type from the typed schema (supports, contradicts, depends_on, repairs, and so on), survive deduplication, and be consistent with the relations already on the same node pair; for example, a dependency and a contradiction between the same two nodes are not both admitted. We also measure the semantic validity of the relations directly: in an audit of 480 sampled relations, 75% were judged valid and 94% at least partly valid, against 13% and 40% for control edges with shuffled relation types.

  1. A contradiction resolved, with node content. A contradiction is a contradicts edge; it counts toward contradiction load (0.65 × the share of unresolved contradiction edges + 0.35 × the share of contradicted nodes without a repair) until a repair is proposed for it.

    Table R20 — One contradiction, before and after its repair

    StepContent
    Contradiction (E021)N003 (Method): attach CLIP language embeddings to Gaussian primitives and refine open-vocabulary queries with hierarchical semantics contradicts N007 (Method): use one global isotropic scale for every Gaussian, tuned once on the training split. Proposed by FeasibilityCritic and accepted by the validator. Contradiction load 0.973.
    Repair (N013)Introduce a per-region scale field with a learned residual, and evaluate it against the global-scale baseline. Proposed by MechanismProposer; evidence attached: Kerbl et al. (2023).
    After the repairE021 resolved; contradiction load 0.973 → 0.896.
  2. Relation validity. We sampled 120 edges of each of four relation types from the logged graphs, plus 120 control edges with shuffled relation types. Two annotators judged each edge from its two node texts as valid, partly valid, or invalid, with disagreements adjudicated.

    Table R21 — Annotation audit of relation validity

    Relation typeEdgesValid / partly valid / invalidPrecision (valid) [95% CI]
    supports12092 / 21 / 70.77 [0.68, 0.83]
    depends_on12099 / 16 / 50.83 [0.75, 0.88]
    contradicts12088 / 25 / 70.73 [0.65, 0.80]
    repairs12083 / 28 / 90.69 [0.60, 0.77]
    All typed relations480362 / 90 / 280.75 [0.71, 0.79]
    Control: relation types shuffled12015 / 33 / 720.13 [0.08, 0.20]

    Inter-annotator agreement is Cohen's κ = 0.73.

Response to Reviewer dicU (3/3)

[W3 Implementation details for reproducibility]

Table R22 specifies the four items the reviewer lists. The teacher's stopping labels are its continue-or-commit decisions on merged post-round graphs, computed from the graph signals and curated as described in Appendix E.3. At deployment, EIG commits when the calibrated commit score reaches 0.50, and not before round 3. Calibration is a bounded additive bias, clipped to ±0.35 on edit scores and ±0.175 on commit scores. The edit and commit losses are optimized in separate updates within each epoch, with weight 1.0 on the commit loss. EIG-NoCalibration removes only the calibration bias. The revision adds this table to the appendix.

  1. Specification.

    Table R22 — Controller specification

    ItemSpecification
    Teacher stopping labelthe heuristic teacher's binary continue-or-commit decision on each merged post-round graph, computed from the graph signals and curated as described in Appendix E.3; 1,725 post-round states (400 commit, 1,325 continue)
    Deployment stoppingcommit when the calibrated commit score reaches the threshold γ_t = 0.50; no commit before round 3; at most 6 rounds
    Calibration boundsadditive bias on the learned scores computed from the graph signals; clipped to ±0.35 on edit scores and to ±0.175 on the commit score
    Loss combinationedit head: grouped slate-ranking loss (softmax cross-entropy over the candidates of each slate); commit head: class-weighted binary cross-entropy (positive-class weight 1.115); the two losses are optimized in separate updates within each epoch (edit batches, then commit batches), with weight 1.0 on the commit loss
    Teacher edit labelone-step simulation of each candidate on the frozen graph using the four graph signals; the candidate that best reduces the dominant deficit is positive and every other candidate in the slate negative; skip is favored only when the graph is already mature
    Runtime fallbackthe heuristic action replaces the learned one when a runtime token is unmapped, a required target node is missing, or a low-gain kind swap would replace a plausible heuristic action
    TrainingAdamW, learning rate 1e-3, batch size 16, 8 epochs; 400 profiled groups (300 train, 100 dev); 5,327 edit states with 63,833 candidate rows
    Encoderall-MiniLM-L6-v2 (384-d), two relation-aware layers, hidden size 128; 648,722 trainable parameters
  2. What EIG-NoCalibration keeps. EIG-NoCalibration removes only the calibration bias. The critic weights, the commit head, the round-3 commit gate, the threshold γ_t, and the runtime fallback are unchanged.

Dear Reviewer dicU,

Thank you again for your detailed review. We have posted point-by-point responses with the new experiments, and uploaded a revised PDF with the changes marked in blue. As the discussion period closes soon, we would be grateful if you could take a look at our responses and let us know whether they address your concerns. We are happy to answer any further questions.

Best regards, The authors

Reviewer_75H6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper's main idea is how multi-agent LLM systems can work together to improve research project ideas and make them more coherent. The authors used a system called EIG, which uses graphs instead of text to improve these ideas. In this graph, each node represents a component of a research idea, and each edge represents relationships such as support, contradiction, and dependency. The system has five specialized agents, controlled by a learned graph critic, to organize improvement decisions. The graph critic includes two heads: an improvement head, which selects an action for each active agent; the selected changes are then merged; and the commit head evaluates the updated graph to decide whether to continue refining or produce the final proposal. Both use a graph encoder, and a heuristic teacher supervises this controller. The authors compared their method with baselines such as SciPIP, AI-Researcher, Graph of Thoughts, Self-Refine, and VirSci. They evaluated EIG on two benchmarks, AI Idea Bench and LiveIdeaBench, with 512 groups. They report that EIG achieves the highest benchmark scores and the best overall ratings.

Critical strengths

The authors evaluate the EIG against several relevant baselines across two benchmarks. They also provide the inference budget, a cross-backbone experiment, and an ablation study. Using both automatic and human evaluation provides stronger evidence than automatic scores alone. For human evaluation, a smaller blinded expert study has conducted on 24 groups. It also favors EIG, with an overall quality score of 4.01/5 compared with 3.50/5 for VirSci.

Critical weaknesses
  • Appendix E.9 presents a teacher–student audit in which the heuristic policy is re-scored on the same logged graph states and candidate slates as the learned controller. While this is a useful control for comparing decision behavior, it is unclear how the same state analysis supports the downstream quality improvements reported in Table 8. Furthermore, the quality scores in Table 8 are identical to the corresponding end-to-end results in Table 4. Could you clarify whether Table 8 represents an independent counterfactual evaluation from matched states or reports the same end-to-end outcomes?

  • Appendix E.8 presents useful ablations and clearly explains which components are held fixed across comparisons. Particularly, the EIG-TextState and EIG-Text comparison controls for the text controller, while EIG-UntypedEdges provides evidence for the importance of relation-type information. My concern is on the comparison between EIG-text and EIG. It is unclear whether the two controllers are comparable in terms of input embeddings, prediction architecture, model capacity, training procedure, and stopping mechanism. Could the authors clarify which of these components are matched across the two controllers?

  • The learned commit mechanism is an important component of EIG, and the EIG-NoCommit ablation provides evidence that learned stopping contributes to final proposal quality. Section 4.3 interprets episodes surviving to later rounds as harder cases, but difficulty is not independently measured. A particularly useful comparison would be between the learned stopping policy and fixed-round policies at comparable average inference costs. I would also appreciate clarification on how many of the 41 episodes terminating in Round 6 satisfied the learned commit criterion versus reaching the maximum-round limit.

  • Finally, Table 13 appears to report five edit decisions per surviving episode in every round, while Table 9 contains fewer audited decisions. Clarifying how you count the active agents and how they selected the data for the teacher–student comparison.

Questions for authors

Please, check these as well.

  • Figure 4’s caption says the distribution is normalized within rounds, but the displayed y-axis and bars show raw action counts.

  • Section 4.3 refers to Appendix H for a latency comparison between frozen-snapshot and sequential execution. However, Appendix H primarily provides a qualitative argument based on parallel execution rather than measured latency.

8 comments

Response to Reviewer 75H6 (1/5)

[W1 Table 8 repeats Table 4; how the same-state analysis supports the downstream gain]

Table 8 is an independent counterfactual evaluation on matched states, and its EIG-Heuristic column should show the matched-state values; the published column repeated the end-to-end values of Table 4 (7.05 / 6.72 / 6.89). On identical states EIG scores 7.41 against 7.01 for the heuristic, +0.39 pooled [0.34, 0.44], and EIG is better on 80% of AI Idea Bench groups and 70% of LiveIdeaBench groups.

  1. Corrected Table 8. The heuristic is the submitted signal heuristic re-scored counterfactually on the exact logged learned graphs and candidate slates, so both policies decide on identical states and differ only in the decision rule. Scores are on the 512 groups, averaged over three seeds; intervals are 95% paired bootstrap intervals over groups (20,000 resamples).

    Table R10 — Table 8, corrected

    BenchmarkGroupsEIG-Heuristic, matched statesEIGDifference [95% CI]
    AIIB2567.247.69+0.45 [0.38, 0.52]
    Live2566.787.12+0.34 [0.26, 0.42]
    Pooled5127.017.41+0.39 [0.34, 0.44]

    Differences are computed before rounding.

  2. Relation to Table 4. The matched-state gap (+0.39) holds the states fixed and varies only the decision rule. The end-to-end gap in Table 4 (+0.52 [0.47, 0.57], 7.41 against 6.89) compares two complete policies, each running its own trajectory. The two are different quantities, and the end-to-end gap is larger.

  3. Revision. Table 8 is replaced by Table R10, and the sentences that cited the old numbers are corrected (Appendix E.10, Section 4.3, and the Limitations paragraph).

Response to Reviewer 75H6 (2/5)

[W2 Whether EIG-Text and EIG are matched in embeddings, architecture, capacity, training, and stopping]

We reran both text-controller ablations with all five components the reviewer lists matched to EIG, so that the only difference is the variable under test: whether the model uses graph structure (the controller's edges in EIG-Text; also the graph state and synthesis in EIG-TextState). The rerun uses the same generator (Qwen3-8B), the same 512 held-out groups, and the same three seeds as the other ablations. With matching, EIG's advantage over the text controller is +0.56 pooled [0.51, 0.60] (+0.73 in the submitted comparison).

  1. What is matched and what is tested. Table R11 lists the five components the reviewer names; the tested variable is the graph edges.

    Table R11 — EIG-Text (rerun) against EIG

    ComponentEIG-TextEIG
    Input embeddingsall-MiniLM-L6-v2all-MiniLM-L6-v2 (same)
    Prediction architectureedit head and commit head on a 128-d encoder outputsame
    Model capacity648,846 trainable parameters648,722 (within 0.02%)
    Training procedurelabels, group split, losses, optimizer, epochssame
    Stopping mechanismlearned commit head, commit thresholds, earliest commit roundsame
    Other settings (generator, roles, runtime, candidate slates, evaluation)same as EIGsame
    Variable under test: graph edgesnot usedtyped relation edges, with relation-specific message passing
  2. How capacity is matched. In EIG each of the two encoder layers holds one 128×128 matrix per relation type (7 types), 114,688 parameters per layer. In the text controller these are replaced by a feed-forward block of width 446 per layer (114,750 parameters with biases), which has no access to edges, giving 229,500 against 229,376 parameters over the two layers. The two models therefore differ by 124 trainable parameters (+0.02%). The embeddings, the heads, and everything outside the two layers are unchanged. Counts are trainable parameters, excluding the frozen MiniLM.

  3. Result. EIG-TextState isolates the state and synthesis representation with the text controller fixed; EIG-Text isolates the controller's use of graph structure with graph state and synthesis fixed. The submitted rows are shown for comparison.

    Table R12 — State and controller ablations (Qwen3-8B, 512 groups, 3 seeds)

    VariantStateControllerSynth.AIIBLiveAvg.
    EIG-TextState, as submittedTextTextText6.53±0.036.33±0.056.43
    EIG-TextState, matched rerunTextTextText6.76±0.056.60±0.046.68
    EIG-Text, as submittedGraphTextGraph6.78±0.096.58±0.086.68
    EIG-Text, matched rerunGraphTextGraph6.93±0.086.77±0.076.85
    EIG (Ours)GraphGraphGraph7.69±0.097.12±0.087.41

    Paired differences over the 512 groups (95% bootstrap intervals): EIG − EIG-Text is +0.76 [0.69, 0.83] on AIIB, +0.35 [0.28, 0.42] on Live, and +0.56 [0.51, 0.60] pooled, against +0.73 for the submitted rows; EIG-Text − EIG-TextState (state and synthesis) is +0.17 [0.11, 0.23] pooled, against +0.25.

  4. Revision. The EIG-TextState and EIG-Text rows of the ablation table are replaced by the matched reruns, and the Section 4.3 and appendix sentences that cite the old contrast (a 0.73-point controller gain, and a text controller that scores no better than random selection) are updated to the matched values.

Response to Reviewer 75H6 (3/5)

[W3 Learned stopping against fixed-round policies at comparable cost, and how the 41 Round-6 episodes ended]

Against fixed-round policies at comparable cost, learned stopping scores higher: fixed 4 and fixed 5 rounds bracket EIG's average cost, and EIG is ahead of them by +0.45 and +0.32 pooled. All 41 episodes that reached Round 6 ended by satisfying the learned commit criterion; none ended at the round limit.

  1. Fixed-round policies at comparable cost. Each fixed-round row is the EIG-NoCommit configuration (learned edit control, no commit head) with the round count set to the stated value, so the rows differ from EIG only in the stopping rule; fixed 5 is the existing EIG-NoCommit ablation. EIG commits at a mean round of 4.18. All rows use 512 groups and three seeds.

    Table R13 — Learned stopping against fixed-round policies

    Stopping policyMean roundsTraced tokensAIIBLiveAvg.
    Fixed 339.66k6.94±0.086.62±0.076.78
    Fixed 4413.09k7.16±0.096.76±0.086.96
    Fixed 5 (EIG-NoCommit)516.62k7.33±0.106.85±0.097.09
    Fixed 6620.26k7.34±0.096.86±0.087.10
    Learned commit (EIG)4.1813.75k7.69±0.097.12±0.087.41

    Tokens are traced generation tokens per sample. Fixed 4 uses 0.66k tokens fewer than EIG and fixed 5 uses 2.87k more, and EIG is ahead of both. EIG exceeds every fixed-round policy in each of the three seeds; fixed 6 gains only 0.01 over fixed 5 at 20.26k tokens.

  2. How episodes ended. The runtime allows at most six rounds and records, for every episode, whether it ended by a commit or by reaching the round limit. A post-round graph is committed when the commit head's calibrated score reaches the round threshold and the runtime guards pass.

    Table R14 — Termination by final round (512 held-out episodes)

    Final roundEpisodesLearned commitReached round limit
    31241240
    42142140
    51331330
    641410
    Total5125120

    The 41 Round-6 episodes (20 AI Idea Bench, 21 LiveIdeaBench) satisfied the commit criterion: their calibrated commit scores at Round 6 have median 0.72 (range 0.52–0.94) against the 0.50 threshold, all 41 are at or above it, the closest by 0.016, and three are within 0.05.

  3. Wording. Section 4.3 will describe the episodes that survive to later rounds as the later-committing tail.

Reviewer_mTBrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Evolving Idea Graphs (EIG), a multi-agent framework for scientific ideation. EIG represents a developing proposal as a persistent structured state graph with typed edges (e.g., support and conflict). In each round, active agents propose graph-edit candidates from the same frozen snapshot. The shared graph encoder feeds an edit head and a commit head. The edit head selects validated role-local candidates and the commit head predicts whether the merged post-round graph should be committed for synthesis. EIG outperforms the evaluated baselines, and ablations support the effectiveness of its main components.

Critical strengths
  1. The formulation of multi-agent ideation as role-specific edits to a persistent, shared idea graph, followed by a learned commit decision, is interesting. This gives agents a common structure for locating and repairing gaps, rather than repeatedly rewriting a whole draft. The ablations indicate that both the graph representation and the learned edit-and-commit control contribute to proposal quality.
Critical weaknesses
  1. The baseline coverage is not sufficiently comprehensive. The evaluation does not include a broader range of recent, closely related approaches, so the results establish gains only over the selected baselines. This limits the evidence for a meaningful advance over the current state-of-the-arts.
  2. The evaluation sample is limited. The automatic evaluation and expert review each cover only a small subset of benchmark instances. The paper does not justify the sample sizes or explain how instances were selected. It reports variation across repeated runs, but not uncertainty across instances, limiting confidence that the gains generalize to the full benchmarks.
Questions for authors
  1. Could the authors provide additional results on a larger sample of benchmark instances and a broader range of recent, closely related methods?
  2. What performance does the two-head critic achieve before training on the same evaluation set?
  3. Why is the two-head critic learnable from the heuristic-curated labels?
  4. Do the reported gains persist with newer models, given that the main evaluation uses Qwen3-8B and Appendix I uses DeepSeek-V3.2 and GPT-4.1-mini?
  5. Would the authors provide further details on how the automatic benchmark evaluation was conducted?
5 comments

Response to Reviewer mTBr (1/4)

[Q1 Larger sample of benchmark instances and a broader set of recent methods (also W1, W2)]

Yes to both. We enlarged the held-out packet from 512 to 1,024 groups (512 per benchmark) and added three recent systems, ResearchAgent (NAACL 2025), CoI-Agent (EMNLP 2025 Findings), and TrustResearcher (WWW 2026). On the 1,024 groups EIG leads all nine other systems on both benchmarks; the closest, TrustResearcher, trails by 0.28 pooled. In a blinded expert study of all ten systems, EIG has the best score on every rubric dimension and the best rank.

  1. Larger packet and selection rule. The 1,024 groups extend the submitted 512, and none of them is in the 400-group graph-critic corpus. Groups are drawn with a fixed seed (2027): LiveIdeaBench proportionally to domain size across its 22 domains, AI Idea Bench 2025 systematically over its release index. Main results use all 1,024 groups; the ablations in the other responses use the submitted 512.

    Table R1 — Packet composition

    BenchmarkReleased sizeSubmittedRevisedCoverage
    LiveIdeaBench1,180 prompts, 22 domains25651243.4%
    AI Idea Bench 20253,495 target papers25651214.6%
    Total4,6755121,02421.9%
  2. Results on 1,024 groups, with instance-level uncertainty. Intervals are 95% bootstrap intervals over groups (20,000 resamples, stratified by domain on LiveIdeaBench), replacing seed-only error bars; each group is scored as the mean of its three runs. With a paired per-group standard deviation of about 0.7, 512 groups per benchmark give 80% power to detect a gap of about 0.11 under a Holm correction over the nine comparisons.

    Table R2 — Automatic results on the 1,024 groups

    MethodVenueAIIBLiveAvg.Tok.
    Direct—5.84 [5.79, 5.89]6.52 [6.47, 6.57]6.18 [6.14, 6.22]1.69k
    Self-RefineNeurIPS'235.94 [5.89, 5.99]6.60 [6.55, 6.65]6.27 [6.23, 6.31]6.01k
    Graph of ThoughtsAAAI'246.22 [6.17, 6.27]6.68 [6.63, 6.73]6.45 [6.41, 6.49]17.79k
    AI-ResearcherICLR'257.08 [7.03, 7.13]6.76 [6.71, 6.81]6.92 [6.88, 6.96]14.81k
    SciPIParXiv'246.92 [6.87, 6.97]6.74 [6.69, 6.79]6.83 [6.79, 6.87]16.42k
    VirSciACL'256.94 [6.89, 6.99]6.76 [6.71, 6.81]6.85 [6.81, 6.89]14.14k
    ResearchAgentNAACL'257.18 [7.13, 7.23]6.88 [6.83, 6.93]7.03 [6.99, 7.07]14.80k
    CoI-AgentEMNLP'25 Findings7.14 [7.09, 7.20]6.86 [6.81, 6.91]7.00 [6.96, 7.04]15.40k
    TrustResearcherWWW'267.28 [7.22, 7.34]6.92 [6.87, 6.97]7.10 [7.06, 7.14]16.10k
    EIG (Ours)—7.66 [7.61, 7.72]7.10 [7.05, 7.15]7.38 [7.34, 7.42]13.75k

    Tok. is the mean traced generation tokens per sample. Every system other than Direct and Self-Refine uses at least EIG's tokens.

Response to Reviewer mTBr (2/4)

[Q1 (continued)]

  1. Three recent systems under the same interface. We added systems that run from the benchmark-visible packet without retrieval EIG does not also receive, have public code, are multi-stage or multi-agent, and were published at a top venue in 2025–2026. Each is scaled to at least EIG's traced tokens through its own native inference, as for AI-Researcher, SciPIP, and VirSci. ResearchAgent's knowledge store is built from the packet's motivating papers with live literature search disabled; CoI-Agent's idea-chain expansion is restricted to a fixed local corpus built from the packet; TrustResearcher's knowledge curation runs over the packet only.

    Table R3 — Paired contrasts against EIG

    ComparisonΔ AIIBΔ LiveΔ pooled [95% CI]
    EIG − ResearchAgent+0.48+0.22+0.35 [0.31, 0.39]
    EIG − CoI-Agent+0.52+0.24+0.38 [0.34, 0.42]
    EIG − TrustResearcher+0.38+0.18+0.28 [0.24, 0.32]
    EIG − AI-Researcher (strongest earlier baseline)+0.58+0.34+0.46 [0.42, 0.50]

    All nine contrasts are positive on both benchmarks, with Holm-adjusted p ≤ 0.0005.

  2. Blinded human evaluation of all ten systems. All ten systems are rated afresh in one blinded round on 24 held-out groups (12 per benchmark), three reviewers per group, because within-group ranks depend on the method set. Rubric scores are on 1–5; rank is the mean within-group rank among ten systems (lower is better).

    Table R4 — Blinded expert evaluation

    MethodNoveltySignif.Feas.ClarityContextOverallRank
    Direct2.652.432.862.992.432.688.97
    Self-Refine2.862.652.923.072.792.768.17
    Graph of Thoughts3.032.882.973.182.932.967.24
    AI-Researcher3.333.043.293.403.173.185.62
    SciPIP3.192.973.123.253.103.037.03
    VirSci3.573.333.383.543.433.474.53
    ResearchAgent3.673.563.503.563.543.573.64
    CoI-Agent3.443.283.213.423.333.295.12
    TrustResearcher3.813.653.533.693.573.693.14
    EIG (Ours)4.113.963.944.034.114.011.54

    Table R4b — Paired differences, EIG minus system (positive favours EIG; 95% bootstrap intervals over the 24 groups)

    ComparisonΔ Overall [95% CI]Δ Rank [95% CI]
    EIG − TrustResearcher+0.32 [0.18, 0.46]+1.60 [0.94, 2.26]
    EIG − ResearchAgent+0.44 [0.29, 0.60]+2.10 [1.75, 2.43]
    EIG − VirSci+0.54 [0.42, 0.68]+2.99 [2.44, 3.53]
    EIG − CoI-Agent+0.72 [0.58, 0.88]+3.58 [2.94, 4.15]

[Q2 Performance of the two-head critic before training]

The identical two-head critic at random initialization, with no supervision, scores 6.55 without calibration and 6.70 as deployed, against 7.28 and 7.41 after training. With the runtime held fixed, training adds 0.73 and 0.71 pooled points.

  1. Protocol and results. We ran the critic with random weights through the same runtime on the same 512 held-out groups, with the same three seeds as the other ablations (each seed draws its own initialization), in two configurations that match a trained row: without calibration (the EIG-NoCalibration setting) and as deployed. This differs from EIG-Random, which samples legal actions uniformly.

    Table R5 — The critic before and after training (512 groups, 3 seeds)

    ControllerSupervisionAIIBLiveAvg.
    EIG-Randomnone; uniform over legal actions6.77±0.106.60±0.096.69
    Untrained critic, no calibrationnone; random init6.62±0.126.48±0.106.55
    EIG-Heuristicteacher rule7.05±0.086.72±0.076.89
    Untrained critic, as deployednone; random init6.76±0.086.64±0.086.70
    EIG-NoCalibrationlearned7.58±0.096.98±0.097.28
    EIG (Ours)learned7.69±0.097.12±0.087.41
  2. Behaviour of the untrained critic. Without calibration, its most frequent action kind (39–55% of decisions) differs by seed, and it stops at about round 3 or not at all, whereas trained EIG spreads commits over rounds 3–6 (mean 4.18).

    Table R6 — Per-seed behaviour of the untrained critic

    ConfigurationSeedDominant action (share)Mean commit roundAvg.
    No calibration1contradiction (51%)3.056.62
    2skip (55%)no commits; all episodes reach the round limit6.42
    3repair (39%)3.076.61
    As deployed1contradiction (29%)4.096.79
    2skip (27%)4.226.64
    3repair (35%)4.186.67

Response to Reviewer mTBr (3/4)

[Q3 Why the critic is learnable from the heuristic-curated labels]

The labels are computed from the same graph state that the critic reads, and the critic recovers them on held-out groups and improves steadily with more supervision.

  1. The labels are computed from the critic's input. For each candidate, the teacher simulates the edit on the frozen graph, recomputes four graph signals from the visible state, and labels the best candidate positive; each merged post-round graph receives a continue-or-commit label. Supervision is dense (5,327 edit states and 63,833 candidate rows; 1,725 commit states) and split by group (300 train, 100 dev), so the critic cannot succeed by recognizing groups.

  2. Agreement with the teacher on held-out groups. The trained network alone, on the 100 held-out dev groups, before calibration, fallback, and safeguards.

    Table R7 — Held-out imitation of the teacher

    HeadMetricCritic [95% CI]ChancePrior
    EditTop-1 agreement0.607 [0.587, 0.628]0.0830.220
    EditTeacher's choice in top 30.847 [0.829, 0.865]0.2480.550
    EditMean reciprocal rank0.729 [0.712, 0.745]0.2550.370
    CommitAUC0.936 [0.918, 0.953]0.500.650
    CommitF1 (positive class)0.771 [0.730, 0.812]0.380.500

    Pooled over three training seeds. Chance is uniform over the slate (edit) and always-commit (F1). Prior uses the most frequent teacher action for the role and round (edit) and the commit rate by round (commit).

  3. Quality grows with the amount of supervision. We train on nested subsets of the 400 profiled groups, stratified by benchmark, with all other settings fixed, and evaluate without calibration on the 512 held-out groups with three seeds. Quality passes the teacher rule between 40 and 100 groups, with smaller gains at higher doses.

    Table R8 — Supervision and downstream quality

    Training groupsAIIBLiveAvg.Δ vs teacher rule
    0 (untrained critic)6.62±0.126.48±0.106.55−0.34
    40 (10%)6.88±0.116.72±0.106.80−0.09
    100 (25%)7.16±0.126.94±0.077.05+0.16
    200 (50%)7.42±0.146.98±0.067.20+0.31
    400 (100%, EIG-NoCalibration)7.58±0.096.98±0.097.28+0.39
    Teacher rule (EIG-Heuristic)7.05±0.086.72±0.076.89—

[Q4 Whether the gains persist with newer generators]

Yes. EIG leads on all five generators we tested, including the newer Qwen3.8-27B and DeepSeek-V4.1-Flash-1017, with margins of +0.22 to +0.60 over AI-Researcher.

  1. Protocol. We ran Direct, AI-Researcher, and EIG on each backbone with thinking mode off, under the same benchmark-visible inputs, output schema, and three seeds, on the 512 groups used for the existing backbone check. AI-Researcher is budget-matched on each backbone, and the graph critic is the one trained on Qwen3-8B traces, with no retraining per generator.

  2. Results. EIG is ahead of AI-Researcher on both benchmarks on every backbone, and the pooled 95% bootstrap intervals over the 512 groups exclude zero; it is ahead of Direct by +0.87 to +1.28.

    Table R9 — Backbone check (512 groups, 3 seeds, thinking off)

    BackboneMethodAIIBLiveAvg.EIG − AI-Researcher [95% CI]
    Qwen3-8B (main)Direct5.886.566.22
    AI-Researcher7.106.806.95
    EIG (Ours)7.697.127.41+0.46 [0.40, 0.51]
    DeepSeek-V3.2Direct6.536.946.74
    AI-Researcher7.667.187.42
    EIG (Ours)7.977.317.64+0.22 [0.17, 0.27]
    GPT-4.1-miniDirect5.716.396.05
    AI-Researcher6.886.586.73
    EIG (Ours)7.587.087.33+0.60 [0.55, 0.65]
    Qwen3.8-27BDirect6.446.846.64
    AI-Researcher7.367.047.20
    EIG (Ours)7.847.287.56+0.36 [0.31, 0.41]
    DeepSeek-V4.1-Flash-1017Direct6.556.976.76
    AI-Researcher7.607.207.40
    EIG (Ours)7.947.327.63+0.23 [0.18, 0.28]

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal and Revision

Dear Area Chair and Reviewers,

Thank you for the reviews. Below is a short summary of what we added in response and where it appears in the revised PDF (changes in blue). Full answers are in the individual threads.

Evidence added

  1. Larger evaluation and three more baselines (mTBr). The main table now covers 1,024 groups (512 per benchmark) and adds ResearchAgent, CoI-Agent, and TrustResearcher. EIG scores 7.38 pooled [7.34, 7.42]; the closest system, TrustResearcher, scores 7.10. In the ten-system blinded expert study, EIG scores 4.01 overall against 3.69 for TrustResearcher.
  2. The learned controller (mTBr, 75H6, dicU). We added an untrained-critic control and a supervision-dose experiment. On identical logged states EIG improves on the heuristic teacher by +0.39 [0.34, 0.44]; end to end the gap is +0.52. A text controller matched to EIG in embeddings, architecture, capacity, training, and stopping scores 6.85 against 7.41 for EIG. Fixed-round policies score 6.96 at four rounds (13.09k tokens) and 7.09 at five (16.62k), against 7.41 for learned stopping (13.75k), and all 41 Round-6 episodes meet the commit criterion.
  3. Runtime and relation quality (75H6, dicU). Median per-episode latency is 120.9 s for the frozen-snapshot runtime against 406.7 s for sequential execution. In an audit of 480 sampled relations, 75% were judged valid, against 13% for shuffled-type controls.

Corrections. The EIG-Heuristic column of Table 8 now shows the matched-state values; the Figure 4 caption now says the bars are raw counts. Appendix E.10 states the scope of the audit counts, including the Rounds 3–6 commit agreement (72.5%) and that the 37.7% edit disagreement covers only audited, executed edits, and describes how the Table 8 scores were obtained.

Revised PDF. The main text is 10 pages. The original 512-group main table and seven-system human study are kept in the appendix.

We would be glad to answer any remaining questions.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.