- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Multi-agent scientific ideation often hides unsupported claims and contradictions in text-based collaboration. EIG maintains a typed idea graph and learns a two-head controller for parallel graph edits and stopping. On 512 held-out tasks from AI Idea Bench 2025 and LiveIdeaBench, it reports higher automatic scores than the compared baselines and better overall expert ratings on 24 tasks.
Critical strengths
-
The problem is important and well motivated. Localizing flaws and tracking improvements while building research proposals is a valuable goal. Typed scientific claims and relations in the idea graph maintain visible weaknesses throughout idea evolution to support agent collaboration.
-
The paper is generally clearly written and easy to understand.
-
The evaluation is comprehensive. It includes several ablations, portability checks with two additional generators, and blinded expert evaluation independent of the automatic scorers.
Critical weaknesses
-
Why does the learned controller outperform its heuristic teacher that provides its training labels? The controller is trained to imitate the heuristic teacher, but Table 4 reports average scores of 7.28 without calibration and 7.41 for full EIG, both exceeding the 6.89 achieved by direct heuristic control. Although Appendix E.9 reports that the full learned policy changes 37.7% of edit choices, it does not establish which changes produce better outcomes or how the retained calibration, fallbacks, and safeguards contribute.
-
The semantic validation of graph relations remains unclear. Appendix K provides a useful refinement example and supporting literature, but does not clearly connect that literature to the evidence attached during execution or illustrate how an explicit contradiction is resolved. Could the authors clarify these connections using concrete node content and edits, and explain how relation validity and contradiction resolution are assessed?
-
More implementation details would improve reproducibility. Please provide more details such as the rules for generating teacher stopping labels, deployment stopping thresholds, calibration bounds, and how the training losses are combined. It would also be helpful to clarify which fallback rules and safeguards are retained in NoCalibration.
Questions for authors
See weaknesses
4 comments
Response to Reviewer dicU (2/3)
[W2 Semantic validation of graph relations, and how a contradiction is resolved]
Relations are validated against the idea's semantic structure when they are proposed. A candidate must connect existing idea nodes, use an action and a relation type from the typed schema (supports, contradicts, depends_on, repairs, and so on), survive deduplication, and be consistent with the relations already on the same node pair; for example, a dependency and a contradiction between the same two nodes are not both admitted. We also measure the semantic validity of the relations directly: in an audit of 480 sampled relations, 75% were judged valid and 94% at least partly valid, against 13% and 40% for control edges with shuffled relation types.
-
A contradiction resolved, with node content. A contradiction is a
contradictsedge; it counts toward contradiction load (0.65 × the share of unresolved contradiction edges + 0.35 × the share of contradicted nodes without a repair) until a repair is proposed for it.Table R20 — One contradiction, before and after its repair
Step Content Contradiction (E021) N003 (Method): attach CLIP language embeddings to Gaussian primitives and refine open-vocabulary queries with hierarchical semantics contradicts N007 (Method): use one global isotropic scale for every Gaussian, tuned once on the training split. Proposed by FeasibilityCritic and accepted by the validator. Contradiction load 0.973. Repair (N013) Introduce a per-region scale field with a learned residual, and evaluate it against the global-scale baseline. Proposed by MechanismProposer; evidence attached: Kerbl et al. (2023). After the repair E021 resolved; contradiction load 0.973 → 0.896. -
Relation validity. We sampled 120 edges of each of four relation types from the logged graphs, plus 120 control edges with shuffled relation types. Two annotators judged each edge from its two node texts as valid, partly valid, or invalid, with disagreements adjudicated.
Table R21 — Annotation audit of relation validity
Relation type Edges Valid / partly valid / invalid Precision (valid) [95% CI] supports120 92 / 21 / 7 0.77 [0.68, 0.83] depends_on120 99 / 16 / 5 0.83 [0.75, 0.88] contradicts120 88 / 25 / 7 0.73 [0.65, 0.80] repairs120 83 / 28 / 9 0.69 [0.60, 0.77] All typed relations 480 362 / 90 / 28 0.75 [0.71, 0.79] Control: relation types shuffled 120 15 / 33 / 72 0.13 [0.08, 0.20] Inter-annotator agreement is Cohen's κ = 0.73.
Response to Reviewer dicU (3/3)
[W3 Implementation details for reproducibility]
Table R22 specifies the four items the reviewer lists. The teacher's stopping labels are its continue-or-commit decisions on merged post-round graphs, computed from the graph signals and curated as described in Appendix E.3. At deployment, EIG commits when the calibrated commit score reaches 0.50, and not before round 3. Calibration is a bounded additive bias, clipped to ±0.35 on edit scores and ±0.175 on commit scores. The edit and commit losses are optimized in separate updates within each epoch, with weight 1.0 on the commit loss. EIG-NoCalibration removes only the calibration bias. The revision adds this table to the appendix.
-
Specification.
Table R22 — Controller specification
Item Specification Teacher stopping label the heuristic teacher's binary continue-or-commit decision on each merged post-round graph, computed from the graph signals and curated as described in Appendix E.3; 1,725 post-round states (400 commit, 1,325 continue) Deployment stopping commit when the calibrated commit score reaches the threshold γ_t = 0.50; no commit before round 3; at most 6 rounds Calibration bounds additive bias on the learned scores computed from the graph signals; clipped to ±0.35 on edit scores and to ±0.175 on the commit score Loss combination edit head: grouped slate-ranking loss (softmax cross-entropy over the candidates of each slate); commit head: class-weighted binary cross-entropy (positive-class weight 1.115); the two losses are optimized in separate updates within each epoch (edit batches, then commit batches), with weight 1.0 on the commit loss Teacher edit label one-step simulation of each candidate on the frozen graph using the four graph signals; the candidate that best reduces the dominant deficit is positive and every other candidate in the slate negative; skipis favored only when the graph is already matureRuntime fallback the heuristic action replaces the learned one when a runtime token is unmapped, a required target node is missing, or a low-gain kind swap would replace a plausible heuristic action Training AdamW, learning rate 1e-3, batch size 16, 8 epochs; 400 profiled groups (300 train, 100 dev); 5,327 edit states with 63,833 candidate rows Encoder all-MiniLM-L6-v2 (384-d), two relation-aware layers, hidden size 128; 648,722 trainable parameters -
What EIG-NoCalibration keeps. EIG-NoCalibration removes only the calibration bias. The critic weights, the commit head, the round-3 commit gate, the threshold γ_t, and the runtime fallback are unchanged.
Dear Reviewer dicU,
Thank you again for your detailed review. We have posted point-by-point responses with the new experiments, and uploaded a revised PDF with the changes marked in blue. As the discussion period closes soon, we would be grateful if you could take a look at our responses and let us know whether they address your concerns. We are happy to answer any further questions.
Best regards, The authors
Response to Reviewer dicU (1/3)
[W1 Why the learned controller outperforms its heuristic teacher, and how calibration, fallback, and safeguards contribute]
The critic is trained on the teacher's within-slate preferences and learns beyond them. The teacher labels the best candidate in each slate from a one-step simulation on four graph signals, while the critic scores candidates from the full graph, the target node's neighbourhood, and the candidate and state text. The trained critic agrees with the teacher's choice on 60.7% of held-out edit decisions, and the final policy departs from the heuristic on 37.7% of edits on identical states. EIG beats the heuristic by +0.39 pooled on identical states and +0.52 end to end, and calibration adds a further +0.13.
Gap to the teacher, two ways. On identical logged states and candidate slates, the two policies differ only in the decision rule (Table R10, response to Reviewer 75H6). End to end, each policy runs its own trajectory (Table 4). Intervals are 95% paired bootstrap intervals over groups; the learned policy's quality passes the teacher's with 100 of the 400 training groups (Table R8, response to Reviewer mTBr).
Table R17 — EIG minus the heuristic teacher (512 groups, 3 seeds)
Where the gain is concentrated. For each of the 512 groups, we take the share of its audited edit decisions on which EIG differs from the heuristic, and the group's matched-state score gap. Groups with at least one differing edit are split into thirds by that share.
Table R18 — Matched-state gap by how much EIG departs from the heuristic
Groups in which no audited edit differs have a gap near zero (+0.01), and the gap grows with the share of edits on which the two policies differ.
Contribution of each component. Replacing learned edit control by the heuristic gives 6.94 (−0.47 against EIG), and replacing learned commit control by a fixed horizon gives 7.09 (−0.32). The ladder below adds the runtime mechanisms; all rows are on the 512 groups.
Table R19 — From the teacher rule to the full system
Calibration contributes +0.13 (+0.11 AIIB, +0.14 Live). The learned critic scores above the teacher (7.01 against 6.89) even with the phase mask, fallback, calibration, and safeguards removed, and adding the mask, fallback, and safeguards brings it to 7.28.