- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SciGraph, a graph-based pipeline for generating and evaluating research ideas. Then the authors evaluate the effectiveness of their proposed evaluator, where the ground truth for the evaluation of each dimension is "first combination ratio", "citation influence of the literature relied on" and "real execution results", respectively.
Critical strengths
- The method is transparent and trageted to be explanable, which I think is significant in scientifc idea evaluation.
- The evaluation ground truth in this paper is interesting, especially effectiveness: using CodeBuddy to execute the candidate idea and compare the result with task-specific baselines under fixed experimental protocols, which provides a new direction to create ground-truth label of ideas.
Critical weaknesses
- Concern about the validity of ground truth label construction methods for novelty and significance. Actually, "first combination ratio" is a very early-stage proxy label of novelty, which can't represent the real novelty score given by humans. And the citation of reference literature is also a proxy label of significance. Also, for effectiveness, I think the ground truth is based on a hypothesis that CodeBuddy could perfectly execute every idea.
- Concern about novelty. I've also searched for some similar work using graphs or literature as grounded evidence in idea evaluation, like ScholarEval and DeepInstructor (but the authors didn't cite them).
Questions for authors
Although the author stated human labels are expensive, there are some open-source datasets like the one in the paper "Can LLMs Generate Novel Research Ideas? ...". Does the evaluator perform well on it? I suggest the author can use a model with a knowledge cut-off date before this work.
3 comments
Thanks author for the clarifications on my questions. However, I am still not convinced on Q1 as Qwen2.5-14B + other baselines (e.g. RAG) is not provided to exclude the improvements from external source of information.
Follow-up on Q1
Follow-up on Q1: external information versus graph scoring. To address whether our gains arise simply from external sources of information, we added two RAG baselines (BM25 and BGE dense retrieval) and a SciGraph-guided paper + LLM evaluation control without graph scoring.
All use Qwen2.5-14B-Instruct and the same 200-paper corpus across seven topics, restricted to papers published before 2024-06-30. We evaluate the same AI-generated, human-written, and AI-reranked conditions (49 ideas each) against the same mean researcher novelty ratings. All three added controls pass each idea and the top-five papers' titles and abstracts to the same judging prompt; paper selection uses BM25, bge-small cosine similarity, or SciGraph mechanism-family overlap, respectively. The table retains the two original methods and adds the three controls.
The SciGraph-guided control directly addresses the information-source concern. It shares SciGraph's mechanism extraction and family mapping, selects five papers by mechanism-family overlap, and asks the LLM to judge novelty from their titles and abstracts instead of computing Nov. Thus, it retains mechanism-guided paper selection while changing the evaluation procedure.
| Method | AI τ | p | Human τ | p | AI-reranked τ | p |
|---|---|---|---|---|---|---|
| Qwen2.5-14B (direct) | −0.075 | 0.45 | +0.065 | 0.54 | +0.005 | 0.99 |
| Qwen2.5-14B + BM25 RAG | +0.139 | 0.157 | +0.335 | <0.001 | +0.085 | 0.388 |
| Qwen2.5-14B + dense RAG (BGE) | +0.202 | 0.041 | +0.193 | 0.058 | +0.237 | 0.016 |
| Qwen2.5-14B + SciGraph-guided papers + LLM judge (no graph scoring) | +0.210 | 0.036 | +0.400 | <0.001 | +0.152 | 0.132 |
| Qwen2.5-14B + SciGraph | +0.247 | 0.012 | +0.429 | <0.001 | +0.389 | <0.001 |
SciGraph ranks first in all three conditions, ahead of both RAG baselines and the SciGraph-guided LLM control, with positive correlations significant in every condition. RAG improves over direct evaluation, but neither adding retrieved literature nor passing SciGraph-selected papers' titles and abstracts to the LLM reproduces the full method's agreement. The SciGraph-guided comparison supports a benefit from the structured evaluation pipeline beyond paper identification alone.
Why graph scoring helps. Retrieval supplies relevant papers; it does not itself determine whether a proposed mechanism combination has historical precedent. SciGraph resolves mentions to canonical mechanism families and computes Nov from historical combination-edge evidence. This makes the comparison operate on mechanisms and their combinations rather than leaving the judge to infer novelty from the overall similarity of prose.
Additionally, we observe that BM25 RAG agrees better with human novelty ratings than dense RAG on the Human subset (τ = 0.335 vs. 0.193). This likely reflects BM25's ability to retrieve exact subtechnique terms, whereas dense retrieval can favor broadly similar topics and miss method-specific precedents. This suggests that novelty in these human-written ideas is often expressed as a specific mechanism-level increment, not a change of topic. SciGraph explicitly represents mechanisms and their historical combinations, matching this form of novelty and achieving the highest agreement on Human (τ = 0.429). We will add these comparisons and discussion to the revision.
[W1] Reference labels and the execution assumption.
Novelty and importance. We agree that FracFirst and citation Grounding are bibliometric proxies, and we note that the paper does not treat them as ground truth: Sec. 3.3, Sec. 5, and App. B state that novelty and importance "have no physical ground truth" for unpublished ideas and use these indicators as external reference orders. Calibrated against ex-post citation percentiles, they reach τb = 0.35/0.42 (1-/2-year windows) for FracFirst and 0.21–0.38 for Grounding (App. B.4). We will state this more prominently in the main text. For agreement with human novelty ratings, please see Q1. Effectiveness. Our label is the observed outcome of one implementation under a fixed agent, budget, protocol, and baseline, not the intrinsic effectiveness of the abstract idea. We do not assume every idea becomes an improvement: the original probe keeps all attempts (46 above, 11 tie, 43 below), and the coding agent fails on 11 of 100 and 143 of 1,000 external candidates (General Response). All generated code was executed; fidelity, i.e. whether a running implementation realizes the idea, was checked manually on 2 implementations per domain of the original probe (20 in total). All 20 correctly implement the stated mechanisms (95% upper bound on the error rate ≈ 14%), likely because code synthesis draws on the real repositories of the carrier papers. Execution quality can still be confounded with idea quality; we will state this limitation and rename the "effectiveness" label "observed execution gain under the specified pipeline".
[W2] ScholarEval and DeepInstructor.
Thank you for pointing out these omissions. We read both papers and will cite and discuss them. The closest overlap is ScholarEval's soundness module, which, like our Val, asks whether similar methods in the literature succeeded.
The key distinction is not "graph vs. no graph": both systems are validated against human or review judgments, whereas we score ideas before implementation and check the scores against execution outcomes. ScholarEval's evidence is unstructured text synthesized by an LLM, and DeepInstructor's graph encodes reviewers' opinions, whereas ours encodes papers' empirical results. We view these approaches as complementary and will avoid overstating our novelty.
[Q1] Si et al. human-rated benchmark.
Thank you for this suggestion. We built a pre-study graph from 200 papers across the benchmark's 7 topics, all published before 2024-06-30, and evaluated all three conditions (AI-generated, human-written, AI-reranked; 49 ideas each), averaging researcher ratings per idea. Mechanisms were extracted with Qwen2.5-14B-Instruct (knowledge cutoff 2024-06) and resolved to the graph's mechanism families, and ideas were scored with Nov. As a same-backbone baseline, Qwen2.5-14B rates novelty directly from the same literature.
With the same backbone, direct evaluation shows no agreement with human novelty ratings, whereas SciGraph agrees significantly in every condition. Agreement is highest on human-written ideas, likely because they state their increment over existing work explicitly, mostly as new mechanisms or mechanism combinations; reviewers judge novelty by this increment, and it is what Nov measures. We will add this experiment to the main text.