acceptodds
Under review as a conference paper at ICLR 2027

From Black-Box Judgments to Auditable Graphs: Structured Idea Generation and Evaluation via Scientific Knowledge Graphs

Abstract

Recent advances in large language models (LLMs) have fueled growing interest in autonomous research-idea generation. Now that many systems can propose ideas, candidates are no longer hard to come by; what is hard is deciding which of them justify the cost of implementation. This shifts attention to idea evaluators and raises a more fundamental question: how should these evaluators themselves be evaluated? Existing benchmarks rely on two unreliable signals: black-box LLM judgments that are unauditable and prompt-sensitive, and proxy labels, such as peer-review scores, publication time, and expert annotations, that are validated against human agreement rather than against actual implementation outcomes. We present SciGraph, a graph-based pipeline for generating and evaluating research ideas. SciGraph decomposes historical literature into meta, mechanism, and protocol nodes and connects them through citation, mechanism-combination, and protocol-result comparison relations. Candidate ideas are randomly sampled from a predefined space and rendered as text. Two fixed policies—mainstream and novelty-oriented—then select candidates for execution, differing only in how they weight three graph-derived priors. To evaluate an idea, SciGraph maps its text back to graph nodes and scores it using the same priors. The pipeline also yields SciGraph-Probe, a probe set for assessing idea evaluators. It contains auditable graph representations of 184 papers across 10 agent-related domains, comprising 184 meta nodes, 490 mechanism nodes, and 727 protocol records connected by more than 1,700 edges. It also includes 100 randomly sampled candidate ideas that undergo code synthesis and execution, producing effectiveness labels withheld from all evaluators, together with external bibliometric reference orders for novelty and importance. We conduct two separate experiments. On three idea-generation benchmarks, using the context provided by each benchmark, the two policies show the expected differences. On SciGraph-Probe, SciGraph ranks first on all three dimensions—effectiveness, novelty, and importance—and every score it produces is traceable to individual nodes and edges.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
auto research, idea generation, idea evaluation, scientific graph.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nU7jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SciGraph, a graph-based pipeline for generating and evaluating research ideas. Then the authors evaluate the effectiveness of their proposed evaluator, where the ground truth for the evaluation of each dimension is "first combination ratio", "citation influence of the literature relied on" and "real execution results", respectively.

Critical strengths
  1. The method is transparent and trageted to be explanable, which I think is significant in scientifc idea evaluation.
  2. The evaluation ground truth in this paper is interesting, especially effectiveness: using CodeBuddy to execute the candidate idea and compare the result with task-specific baselines under fixed experimental protocols, which provides a new direction to create ground-truth label of ideas.
Critical weaknesses
  1. Concern about the validity of ground truth label construction methods for novelty and significance. Actually, "first combination ratio" is a very early-stage proxy label of novelty, which can't represent the real novelty score given by humans. And the citation of reference literature is also a proxy label of significance. Also, for effectiveness, I think the ground truth is based on a hypothesis that CodeBuddy could perfectly execute every idea.
  2. Concern about novelty. I've also searched for some similar work using graphs or literature as grounded evidence in idea evaluation, like ScholarEval and DeepInstructor (but the authors didn't cite them).
Questions for authors

Although the author stated human labels are expensive, there are some open-source datasets like the one in the paper "Can LLMs Generate Novel Research Ideas? ...". Does the evaluator perform well on it? I suggest the author can use a model with a knowledge cut-off date before this work.

3 comments

[W1] Reference labels and the execution assumption.

Novelty and importance. We agree that FracFirst and citation Grounding are bibliometric proxies, and we note that the paper does not treat them as ground truth: Sec. 3.3, Sec. 5, and App. B state that novelty and importance "have no physical ground truth" for unpublished ideas and use these indicators as external reference orders. Calibrated against ex-post citation percentiles, they reach τb = 0.35/0.42 (1-/2-year windows) for FracFirst and 0.21–0.38 for Grounding (App. B.4). We will state this more prominently in the main text. For agreement with human novelty ratings, please see Q1. Effectiveness. Our label is the observed outcome of one implementation under a fixed agent, budget, protocol, and baseline, not the intrinsic effectiveness of the abstract idea. We do not assume every idea becomes an improvement: the original probe keeps all attempts (46 above, 11 tie, 43 below), and the coding agent fails on 11 of 100 and 143 of 1,000 external candidates (General Response). All generated code was executed; fidelity, i.e. whether a running implementation realizes the idea, was checked manually on 2 implementations per domain of the original probe (20 in total). All 20 correctly implement the stated mechanisms (95% upper bound on the error rate ≈ 14%), likely because code synthesis draws on the real repositories of the carrier papers. Execution quality can still be confounded with idea quality; we will state this limitation and rename the "effectiveness" label "observed execution gain under the specified pipeline".

[W2] ScholarEval and DeepInstructor.

Thank you for pointing out these omissions. We read both papers and will cite and discuss them. The closest overlap is ScholarEval's soundness module, which, like our Val, asks whether similar methods in the literature succeeded.

ScholarEvalDeepInstructorSciGraph
EvidencePapers retrieved per idea from Semantic ScholarExperience Graph of 416K triples from 58,607 ICLR reviewsGraph of mechanisms, protocols with numerical results, and citations from the reference papers
ScoringLLM-written review with citationsReAct agent writes comments; few-shot LLM scorer maps them to 1–10Deterministic graph calculation given the resolved mapping
DimensionsSoundness, contributionNovelty, significance, feasibilityEffectiveness, novelty, importance
Validated againstLLM-judged coverage of expert review rubrics (117 ideas); user studyHuman scores on Si et al. data; ICLR-review tiersExecution outcomes; bibliometric reference orders

The key distinction is not "graph vs. no graph": both systems are validated against human or review judgments, whereas we score ideas before implementation and check the scores against execution outcomes. ScholarEval's evidence is unstructured text synthesized by an LLM, and DeepInstructor's graph encodes reviewers' opinions, whereas ours encodes papers' empirical results. We view these approaches as complementary and will avoid overstating our novelty.

[Q1] Si et al. human-rated benchmark.

Thank you for this suggestion. We built a pre-study graph from 200 papers across the benchmark's 7 topics, all published before 2024-06-30, and evaluated all three conditions (AI-generated, human-written, AI-reranked; 49 ideas each), averaging researcher ratings per idea. Mechanisms were extracted with Qwen2.5-14B-Instruct (knowledge cutoff 2024-06) and resolved to the graph's mechanism families, and ideas were scored with Nov. As a same-backbone baseline, Qwen2.5-14B rates novelty directly from the same literature.

MethodAI τpHuman τpAI-reranked τp
Qwen2.5-14B (direct)−0.0750.45+0.0650.54+0.0050.99
Qwen2.5-14B + SciGraph+0.2470.012+0.429<0.001+0.389<0.001

With the same backbone, direct evaluation shows no agreement with human novelty ratings, whereas SciGraph agrees significantly in every condition. Agreement is highest on human-written ideas, likely because they state their increment over existing work explicitly, mostly as new mechanisms or mechanism combinations; reviewers judge novelty by this increment, and it is what Nov measures. We will add this experiment to the main text.

Thanks author for the clarifications on my questions. However, I am still not convinced on Q1 as Qwen2.5-14B + other baselines (e.g. RAG) is not provided to exclude the improvements from external source of information.

AuthorsOpenReview ↗

Follow-up on Q1

Follow-up on Q1: external information versus graph scoring. To address whether our gains arise simply from external sources of information, we added two RAG baselines (BM25 and BGE dense retrieval) and a SciGraph-guided paper + LLM evaluation control without graph scoring.

All use Qwen2.5-14B-Instruct and the same 200-paper corpus across seven topics, restricted to papers published before 2024-06-30. We evaluate the same AI-generated, human-written, and AI-reranked conditions (49 ideas each) against the same mean researcher novelty ratings. All three added controls pass each idea and the top-five papers' titles and abstracts to the same judging prompt; paper selection uses BM25, bge-small cosine similarity, or SciGraph mechanism-family overlap, respectively. The table retains the two original methods and adds the three controls.

The SciGraph-guided control directly addresses the information-source concern. It shares SciGraph's mechanism extraction and family mapping, selects five papers by mechanism-family overlap, and asks the LLM to judge novelty from their titles and abstracts instead of computing Nov. Thus, it retains mechanism-guided paper selection while changing the evaluation procedure.

MethodAI τpHuman τpAI-reranked τp
Qwen2.5-14B (direct)−0.0750.45+0.0650.54+0.0050.99
Qwen2.5-14B + BM25 RAG+0.1390.157+0.335<0.001+0.0850.388
Qwen2.5-14B + dense RAG (BGE)+0.2020.041+0.1930.058+0.2370.016
Qwen2.5-14B + SciGraph-guided papers + LLM judge (no graph scoring)+0.2100.036+0.400<0.001+0.1520.132
Qwen2.5-14B + SciGraph+0.2470.012+0.429<0.001+0.389<0.001

SciGraph ranks first in all three conditions, ahead of both RAG baselines and the SciGraph-guided LLM control, with positive correlations significant in every condition. RAG improves over direct evaluation, but neither adding retrieved literature nor passing SciGraph-selected papers' titles and abstracts to the LLM reproduces the full method's agreement. The SciGraph-guided comparison supports a benefit from the structured evaluation pipeline beyond paper identification alone.

Why graph scoring helps. Retrieval supplies relevant papers; it does not itself determine whether a proposed mechanism combination has historical precedent. SciGraph resolves mentions to canonical mechanism families and computes Nov from historical combination-edge evidence. This makes the comparison operate on mechanisms and their combinations rather than leaving the judge to infer novelty from the overall similarity of prose.

Additionally, we observe that BM25 RAG agrees better with human novelty ratings than dense RAG on the Human subset (τ = 0.335 vs. 0.193). This likely reflects BM25's ability to retrieve exact subtechnique terms, whereas dense retrieval can favor broadly similar topics and miss method-specific precedents. This suggests that novelty in these human-written ideas is often expressed as a specific mechanism-level increment, not a change of topic. SciGraph explicitly represents mechanisms and their historical combinations, matching this form of novelty and achieving the highest agreement on Human (τ = 0.429). We will add these comparisons and discussion to the revision.

Reviewer_Mt2FOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents SciGraph, which represents reference literature with paper-metadata, mechanism, and experimental-protocol nodes connected by citation, mechanism-combination, and result-comparison relations. Candidate ideas are sampled from a mechanism–task–protocol space, rendered as text, and selected using fixed mainstream or novelty-oriented policies. The same graph supplies novelty, importance, and validity priors after candidate-to-graph mapping. The paper also introduces SciGraph-Probe and conducts separate experiments on idea generation and evaluator assessment.

Critical strengths
  1. SciGraph-Probe pairs 100 candidate ideas with real execution outcomes and provides evaluation protocols and comparisons of existing evaluators, offering a useful resource for future research on idea evaluation.
  2. Under a shared backbone and benchmark context, the framework improves several generation metrics across IdeaBench-research, AI Idea Bench 2025, and IDRBench I3 (e.g., 82.5% versus 78.1% composition on IDRBench; Table 1).
Critical weaknesses
  1. All 100 probe candidates are produced and selected by SciGraph, with Val used for both selection and effectiveness prediction. This coupling may favor the proposed evaluator and limits conclusions beyond its own candidate distribution.
  2. Val relies on mechanism-set similarity and historical results for the same protocol, while the probe primarily tests recombinations of existing building blocks; performance and coverage on ideas with unseen mechanism families remain unvalidated, e.g. idea of new benchmarks or tasks.
Questions for authors
  1. In Figure 2, the selected candidates appear underrepresented at the high end of the Grounding ranking despite policy selection. Could you quantify this by policy and domain, and compare before/after selection to clarify whether selection improves citation grounding?

  2. How should effectiveness be defined and validated for ideas centered on new benchmarks, datasets, or task formulations, whose value need not appear as improvement on an existing protocol?

2 comments

[W1] Self-selection and Val coupling.

Thank you. To be precise, the candidate texts are not produced by the policies. They are randomly sampled from a predefined mechanism–task–protocol space into a shared pool (N = 30 per domain) and rendered as text; the fixed policies only select which candidates are executed (Sec. 3.2, App. B.3). We agree, however, that the executed set is policy-conditioned and that Val appears in both selection and prediction. To test this coupling, we evaluated candidates from an external generator, The AI Scientist, which uses neither our sampling space nor our policies or priors (General Response, Tables GR-2/GR-3):

  1. Effectiveness. Execution labels are not constructed from any graph quantity, so this directly tests the coupling. On the 857 executed AIS-1000 candidates, Val reaches 0.677 accuracy and 0.725 F1, vs. 0.508 and 0.580 for the LLM judge and a 0.560 majority rate. With a similar share predicted effective, its gain comes from rejecting the right candidates: NPV 0.947 vs. 0.626. AIS-100 shows the same pattern.
  2. Ranking. Novelty/importance τ are +0.30/+0.33 on AIS-100 and +0.31/+0.33 on AIS-1000, vs. +0.28/+0.30 on the original probe; the LLM judge stays at +0.10–0.14 on all three sets. The reference orders are computed outside the graph (raw-paper counts and same-year, same-venue citation percentiles; App. B.1–B.2) and never enter SciGraph.

We will add these results and state in Sec. 3.3 that the original probe is policy-conditioned.

[Q1] Does selection improve Grounding (Figure 2)? Thank you. We compare each policy's executed candidates with the full pre-selection pools ("No policy": 30 random candidates per domain, 300 in total, not executed):

IndicatorNovelty policyMainstream policyNo policy
Mean FracFirst (novelty)0.8840.8280.790
Mean Grounding (importance)0.5900.6090.552
Domains where FracFirst exceeds No policy10/109/10—
Domains where Grounding exceeds No policy8/1010/10—
Share in top 20% of joint Grounding ranking12% (6/50)16% (8/50)7.3% (22/300)
  • Selection raises both indicators, consistently across domains. The novelty policy gives the largest FracFirst gain (+0.094, in 10/10 domains) and the mainstream policy the largest Grounding gain (+0.057, in 10/10 domains); the novelty policy also raises Grounding (+0.038).
  • At the high end, selection also helps: 16% of mainstream and 12% of novelty candidates reach the top 20% of the joint ranking, vs. 7.3% of the pre-selection pools. Both shares remain below 20%, so selected candidates are still underrepresented at the top, as you observed.

Three reasons limit the high end: (1) the selection prior Imp is deliberately not the Grounding reference (it adds a bridging term and uses in-pool citations rather than same-year, same-venue percentiles), which avoids leakage; (2) each policy picks 5 of 30 random candidates and can only reorder that pool; (3) the mainstream policy weights Val as heavily as Imp. We will add this table with per-domain results and revise the Figure 2 discussion.

[W2 & Q2] Unseen mechanisms and new benchmarks/tasks.

We agree this is a limitation. Our effectiveness label is the observed execution gain over a fixed baseline on an existing protocol, and the probe mainly tests recombinations of existing mechanisms. Ideas whose primary contribution is a new benchmark, dataset, or task formulation are therefore outside the scope of this label and are not validated in this work. What SciGraph does on such ideas. Val is reported as unknown, not replaced by a default, when the target protocol has no historical baseline comparison (App. B.3). Unmatched mechanisms are retained as out-of-graph and flagged by Coverage. Low Coverage indicates insufficient historical evidence; a high Nov score then does not by itself establish genuine novelty. How effectiveness could be defined for them. For a new benchmark or dataset, a natural definition is the measurement value it adds: (i) discriminative power, i.e. whether it separates existing methods; (ii) non-redundancy, i.e. low rank correlation with existing benchmarks; and (iii) later adoption. These can be validated by running existing models on the proposed benchmark, or retrospectively on published benchmarks. For a new task formulation, the analogue is whether existing methods transfer to it. We will add this scope statement and discussion to the paper.

Presentation.

Thank you; we will improve readability: (1) split Sec. 3.2, which mixes candidate sampling, policy selection, and scoring, into subsections; (2) add a notation table and remove overloaded symbols (τ is both Kendall's τ and the similarity floor in App. B.3; "Grounding" names both the reference order and an Imp component); (3) state in Sec. 3.2 and Fig. 1, not only Sec. 4.1/App. B.3, that candidates are randomly sampled and policies only select (cf. W1).

Reviewer_Mt2FOpenReview ↗

Response to Authors

I appreciate the additional results and clarify provided by the authors, which address my concerns. Therefore, I keep my positive rating

Reviewer_u5DpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  1. Proposes SciGraph, a heterogeneous graph of 184 papers across 10 AI domains with meta/mechanism/protocol nodes and three views (citation, mechanism co-occurrence, protocol results); two fixed graph-conditional selection policies rank sampled candidates.
  2. Builds a 100-candidate probe actually executed against fixed baselines (46 exceed baseline, 54 below). Evaluation compares four LLM judges, three benchmark-native evaluators, and SciGraph against three external reference orders: real execution outcomes for effectiveness, FracFirst (bibliometric) for novelty, and citation Grounding for importance.
  3. Key findings: LLM judges approach majority-class baseline (acc 0.52–0.60 vs 0.54, recall ~1.0 = call almost everything effective), novelty τ flips sign across backbones; SciGraph ranks first on all three dims (acc 0.70, novelty τ +0.28, importance τ +0.30). Ablations show each view and graph structure matter.
Critical strengths
  • The evaluation protocol is substantially stronger than prior LLM-as-judge work: it pairs unpublished candidates with real execution outcomes and uses external bibliometric reference orders (FracFirst, citation Grounding) kept outside the pipeline, so evaluators cannot be tuned to match them.
  • The graph substrate is genuinely auditable: given LLM-assisted mechanism resolution, every final score is a deterministic graph calculation decomposable into named nodes, edges, and paper rows; this is a meaningful step toward inspectable research-idea judging.
  • Ablations are well-designed — view removal (each degrades primarily its own dimension) and structure controls (edge shuffle / random edges → τ collapses to ~0.10, matching the unstructured judge level) convincingly separate graph signal from generic literature-context benefit.
Critical weaknesses
  • The 184-paper corpus is biased toward AI tasks (Table 7), the 100-candidate probe is small for 10 domains (50 mainstream / 50 novelty, only 46 produce positive execution gains), and the effectiveness label is binary; τ ≈ 0.28–0.30 are weak rank correlations, limiting practical usefulness despite the clean protocol.
  • Several critical hyperparameters are fixed but neither justified nor sensitivity-analyzed: the Val(c, g) ≥ 0.5 effectiveness threshold, the policy weights (0.2/0.4/0.4 vs 0.6/0.2/0.2), and the 0.8/0.2 and 0.4/0.6 splits inside Nov and Imp. Appendix B.3 merely states they are "fixed before execution," with no robustness check.
  • SciGraph still depends on an LLM for the mechanism-resolution step; the "deterministic" claim applies only AFTER that LLM has already mapped each candidate text to mechanism families, and no failure-rate or resolution-quality ablation assesses how final scores depend on getting this right.
Questions for authors
  • What happens to SciGraph scores under sensitivity analysis of the Val threshold, the policy weights, and the Nov/Imp component weights? Did any informal tuning inform the final choices, or are they truly first guesses?
  • What is the failure rate of LLM-assisted mechanism resolution, and can you report how final novelty/importance scores differ when resolution is wrong, incomplete, or leaves mechanisms out-of-graph?
  • Given that τ ≈ 0.3 is a weak ordering, what would convince you that SciGraph's scores are trustworthy for a new, unseen corpus outside AI tasks?
1 comment

[W1 & Q3] Scale, binary label, weak τ, and unseen corpora.

Scale and unseen corpora. We extended the corpus to 20 domains (1,778 papers) and ran 1,000 external candidates from The AI Scientist (857 executed; General Response); on these, novelty/importance τ are +0.31/+0.33, consistent with +0.28/+0.30 on the original probe. The new domains go beyond agent/LLM tasks but mostly remain within ML; corpora outside ML remain to be validated. Practical usefulness. From the same input, SciGraph candidates beat their baselines more often than external ones (46% vs. 38%; 43% of executed runs) and never fail to run (0 vs. 11). This compares candidate sources, not selection rules. Binary label. We agree "effectiveness" is too strong. We mean whether one implementation beats a fixed baseline under a fixed agent, budget, and protocol; ties count as not effective, as in Table 2. We will rename the label "observed execution gain under the specified pipeline" and report the full outcome breakdown and per-candidate gains (Fig. 2, left). τ ≈ 0.3. We agree it is weak (Sec. 4.3, 5), as idea evaluation has no ground truth. Still, SciGraph exceeds both the LLM judges (novelty τ −0.09 to +0.20, importance −0.01 to +0.10) and the benchmark-native evaluators (novelty −0.03 to +0.24, importance +0.05) in Table 2. Edge shuffle and random edges drop τ to ~0.10–0.14 (Table 3), so the signal comes from graph structure. SciGraph is an auditable prior, not an oracle.

[W2 & Q1] Hyperparameters.

These were our first and only settings, fixed before execution; none was tuned on execution labels or reference orders. Val threshold.

ThresholdAccPrecRecF1
0.30.5600.5111.0000.676
0.40.6800.5901.0000.742
0.5 (Table 2)0.7000.6110.9570.746
0.60.7100.6350.8700.734
0.70.6900.6470.7170.680
0.80.6500.6900.4350.533

From 0.4 to 0.7, accuracy stays at 0.68–0.71, above the best LLM judge (0.600); at 0.3, nearly all candidates are called effective, like the LLM judges. 0.6 has slightly higher accuracy but lower F1; we keep the pre-fixed 0.5. Policy weights follow the meaning of the priors: high Nov means a rarer, more likely novel combination; high Imp means mechanisms carried by influential papers that bridge research lines; high Val means similar methods historically beat the baseline. Mainstream thus emphasizes grounding and validity, and novelty rarity. Perturbing the weights changes agreement by ≤ 0.02 within each policy:

Policy (w_nov, w_imp, w_val)Nov. τImp. τ
Mainstream (0.2, 0.4, 0.4)0.300.28
Mainstream var. (0.1, 0.45, 0.45)0.320.27
Novelty (0.6, 0.2, 0.2)0.260.32
Novelty var. (0.8, 0.1, 0.1)0.260.32

Nov/Imp splits. Agreement with the reference orders (Table 2 protocol; defaults in bold):

Nov α0.00.20.40.60.81.0
Nov. τ vs. FracFirst0.020.130.270.290.280.27
Imp β₁0.00.20.40.60.81.0
Imp. τ vs. Grounding0.180.250.300.320.300.29

τ is flat for α, β₁ ∈ [0.4, 0.8], where it is never below either component alone. Defaults are near the maxima (0.29, 0.32 at 0.6). We will add all three analyses.

[W3 & Q2] Mechanism resolution.

The LLM only extracts facts (mechanism mentions matched to canonical families); it produces no score and makes no evaluation decision. Failure rate. For SciGraph-100, the sampled mechanisms give exact ground truth: 250 true mechanisms, 264 extracted.

Match level (cumulative)nRecall (/250)Precision (/264)
Exact string match7831.2%29.5%
+ Semantically equivalent (LLM-judged)20180.4%76.1%
+ Same mechanism family24698.4%93.2%

Since scoring operates on families, 1.6% of true mechanisms are missed (4/250) and 6.8% of extractions are spurious (18/264). These texts are rendered from known mechanisms, the easiest case; free-form ideas are harder. Perturbation. We perturb 10%/20% of resolved mechanisms, at or above these error rates: drop (missed), replace (wrong family), cut edges (isolated node); means of 20 runs.

SettingNov. τ (10% / 20%)Imp. τ (10% / 20%)
Original0.280.30
Drop0.27 / 0.270.26 / 0.27
Replace0.21 / 0.170.24 / 0.23
Cut edges0.26 / 0.240.25 / 0.24

Misidentification hurts most, as a wrong family injects misleading evidence. At 20% replacement, novelty τ (0.17) falls below the strongest baselines (IdeaBench-research 0.24, glm-5.2 0.20), while importance τ (≥ 0.23) stays above all of them. "Deterministic" holds only given the resolved mapping (Sec. 3.2, 4.3); this study, to be added, quantifies that dependence. Mappings are stored with evidence spans for audit.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: external generator, 20 domains, and 1,000 attempted candidates

We thank all reviewers. Several comments (u5Dp W1/Q3, Mt2F W1, nU7j W1) ask whether conclusions drawn from 100 SciGraph-selected candidates over a 184-paper, agent/LLM-centric corpus extend beyond that setting. We ran the following experiment.

Setup.

  • Corpus. We extend the corpus from 10 to 20 domains (1,778 full-text papers), adding GNNs, time-series forecasting, recommender systems, detection/segmentation, 3D vision/NeRF, causal inference, reinforcement learning, speech, foundational NLP, and bioinformatics/drug discovery. These go beyond agent/LLM tasks but mostly remain within ML. Graphs use the same pipeline and hyperparameters.
  • External candidates. All new candidates come from The AI Scientist (Lu et al., 2026, Nature), which uses neither our sampling space nor our policies or priors. AIS-100: 100 candidates from exactly the original probe input (same 10 domains and 184 papers). AIS-1000: 50 per domain across all 20 domains.
  • Protocol. Retry, protocol/baseline assignment, source papers, and τ grouping follow the original probe; evaluator inputs match Table 2. Effectiveness uses successful executions; all 857 on AIS-1000 have defined Val. Ranking uses all 1,000 (20 domains × 50). Failed runs are reported separately, not as ineffective: implementation failure does not imply idea ineffectiveness.

Table GR-1. Execution outcomes.

OutcomeSciGraph-100 (original)AIS-100 (same input)AIS-1000 (20 domains)
Above baseline4638377
Tie11588
Below baseline4346392
Execution failure011143

Table GR-2. Effectiveness classification (ties count as not effective, as in Table 2).

SetEvaluatorAccPrecRecF1
SciGraph-100 (majority 0.54)LLM judge (dsv4f)0.6000.5380.9350.683
SciGraph0.7000.6110.9570.746
AIS-100, 89 executed (majority 0.573)LLM judge (dsv4f)0.5390.4770.8160.602
SciGraph0.6520.5570.8950.687
AIS-1000, 857 executed (majority 0.560)LLM judge (dsv4f)0.5080.4640.7720.580
SciGraph0.6770.5790.9680.725

Table GR-3. Agreement with the external reference orders (protocol of Table 2).

SetEvaluatorNov. τNov. ρImp. τImp. ρ
SciGraph-100LLM judge (dsv4f)+0.12+0.16+0.10+0.13
SciGraph+0.28+0.34+0.30+0.35
AIS-100LLM judge (dsv4f)+0.12+0.18+0.13+0.15
SciGraph+0.30+0.35+0.33+0.37
AIS-1000LLM judge (dsv4f)+0.13+0.19+0.14+0.17
SciGraph+0.31+0.38+0.33+0.38

Findings.

  1. Results hold on candidates SciGraph neither generated nor selected. Execution labels are not constructed from any graph quantity, so effectiveness on AIS candidates directly tests the selection–prediction coupling. On both sets, SciGraph beats the LLM judge on all four metrics; on the 857 executed AIS-1000 candidates, accuracy is 0.677 vs. 0.508 (majority rate 0.560) and F1 0.725 vs. 0.580. Both call a similar share effective (about 73%); the difference is which candidates they reject. 95% of SciGraph's "ineffective" predictions are correct (NPV 0.947 vs. 0.626), and it rejects 45% of truly ineffective candidates vs. 30% for the judge, consistent with App. A.3, where the accuracy gain comes mainly from correctly rejecting candidates. Novelty and importance τ also stay at a similar level, while the LLM judge remains at +0.12–0.14, suggesting that AIS candidates are not simply easier to rank (Table GR-3).
  2. Candidate source, not selection rule. From the same input, SciGraph candidates beat their baselines in 46% vs. 38% of cases (43% of executed runs) and never failed to run (vs. 11%). This compares candidate sources; we still make no claim that our policies beat random selection within a pool (Sec. 4.2, App. B.3). SciGraph candidates come with canonical mechanism families backed by carrier papers and code links, and with protocols that have historical baselines, which lowers the execution barrier; free-form external ideas are more ambitious but more often under-specified. This also reflects conservative synthesis, which we will discuss.
  3. Execution is imperfect. The coding agent fails on 11–14% of external candidates.

Planned revisions.

  • Add these experiments, the domain list, protocol details, and per-domain results.
  • Further emphasize that FracFirst and Grounding are external reference orders (bibliometric proxies), as already stated in Sec. 3.3, Sec. 5, and App. B.
  • Rename "effectiveness" to "observed execution gain over a fixed baseline under the specified pipeline".
  • Add the reviewer-specific analyses below.

Dear Reviewers, thank you for your thoughtful reviews. As the authors have now responded, please review their responses and the other reviews, and leave follow-up comments on whether the concerns have been addressed. Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.