- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents IDEAREWARDBENCH, which comprises 3,185 idea preference instances grounded in multiple complementary signals of idea quality. Evaluating 23 baselines on IDEAREWARDBENCH reveals substantial limitations in existing idea evaluators, highlighting the benchmark’s difficulty and diagnostic value. IDEAREWARD is the first reward model for research idea generation. IDEAREWARD is trained with a fully automated and scalable data construction pipeline, and achieves state-of-the-art performance on IDEAREWARDBENCH.
Critical strengths
This paper has a good struture and can be understandable. present a living benchmark for research idea evaluation and conduct extensive experiments and analyses on IDEAREWARD and evaluated baselines
Critical weaknesses
- Prior work, such as LDC: Learning to Generate Research Idea with Dynamic Control, has already explored training research idea reward models using peer-review feedback, including separate models for novelty, feasibility, and effectiveness. The authors should better clarify what is new beyond the multi-source preference data construction. This related work is also missing from the paper.
- The benchmark uses paper acceptance decisions and citation counts as proxies for idea quality. However, paper acceptance depends heavily on execution and presentation, while citations are influenced by popularity and research trends. It is unclear how reliably these signals reflect the quality of the original research ideas.
Questions for authors
The training and benchmark data are constructed using very similar signals, particularly review scores and citation counts. Although the papers come from different time periods, the model may mainly learn patterns associated with these proxies rather than generalizable research idea quality. More evaluation on independently annotated ideas would strengthen the claims.
The best-of-N experiment only covers 30 topics with seven candidate ideas each. The improvement over GPT-5.5 is also modest (9 wins versus 6 losses, with 15 ties). More importantly, the selected ideas are not actually implemented, making it difficult to tell whether better reward scores translate into more successful research projects.
5 comments
W2 & Q1: Do paper-level proxies reliably reflect idea quality, and does IdeaReward learn generalizable idea quality judgements?
We agree that "paper acceptance depends on execution and presentation, while citations are influenced by popularity and research trends". This is exactly why we combine multi-source signals rather than treating any source as a perfect label, which has been explained in our response to W1. We now respond to the reviewer's two concerns in detail.
(1) Do paper-level preferences reliably reflect idea quality?
To better answer this question, we expanded our pilot study of idea-paper preference agreement from 40 pairs to 200 benchmark instances, comprising 50 instances from each of four subsets: review-grounded pairs, review-grounded quadruples, impact-grounded pairs, and impact-grounded quadruples. Agreement between the human idea-level preferences and the gold paper-level preferences is shown below:
| Split | Agreement between idea-level and paper-level preferences | |
|---|---|---|
| Review-grounded pairs | 50 | 88.0% |
| Review-grounded quadruples | 50 | 84.0% Top-1 Acc / 90.3% PRA |
| Impact-grounded pairs | 50 | 84.0% |
| Impact-grounded quadruples | 50 | 82.0% Top-1 Acc / 92.0% PRA |
| Overall | 200 | 84.5% top-choice agreement |
The results show high agreement overall and across all four subsets, supporting paper-level preferences as informative and scalable proxies for comparative idea quality.
Moreover, deriving idea-quality signals from paper-level outcomes has clear methodological precedence in existing work, including LDC (https://arxiv.org/pdf/2412.14626, EMNLP 2026), as highlighted by the reviewer, and InnoEval (https://arxiv.org/pdf/2602.14367, ICML 2026). This approach is well motivated as the definition of an idea card is essentially the blueprint of a paper: a proposal-stage summary of its motivations, method, and experimental plan. Because a strong underlying idea is a key prerequisite for a strong paper, paper-level outcomes provide an informative and scalable, though inevitably imperfect, proxy for idea quality.
(2) Does IdeaReward learn generalizable idea-quality judgments?
(a) Generalization to independently annotated ideas. Our main results in Section 4.1 and Table 1 show that IdeaReward is the only approach that consistently achieves state-of-the-art performance across all three benchmark splits. In particular, its performance on the annotation-grounded split is especially informative because IdeaReward’s training data contains no human-annotated preferences. This split is therefore out of distribution with respect to the source of preference supervision. Despite receiving no human-annotated training preferences, IdeaReward-Llama and IdeaReward-Qwen achieve 91.5% and 94.0% overall accuracy on this split. On the human-vs-LLM subset, IdeaReward outperforms the second-best evaluator by 15 percentage points. These results provide direct evidence that IdeaReward learns idea-quality judgments that generalize beyond the review- and impact-grounded distributions constituting the majority of its training data.
(b) Ruling out superficial shortcuts. In this analysis, we further show that IdeaReward learns genuine idea quality judgement rather than superficial shortcuts. We compare IdeaReward against three naive heuristic baselines based on idea card length, the card's number of dataset mentions, and the card's number of baseline mentions. The card-length baseline is evaluated on the complete benchmark. The two mention-count baselines are evaluated on 300 instances comprising 50 pairs and 50 quadruples from each review and impact-grounded split, together with all 100 annotation-grounded pairs.
| Evaluator | Review pair acc. | Review quadruple PRA / top-1 | Impact pair acc. | Impact quadruple PRA / top-1 | Annotation pair acc. |
|---|---|---|---|---|---|
| Card length | 50.1% | 52.0% / 27.2% | 48.7% | 47.4% / 24.9% | 55.0% |
| Dataset mentions | 51.0% | 56.7% / 29.0% | 54.0% | 53.8% / 35.0% | 63.0% |
| Baseline mentions | 57.0% | 53.8% / 33.8% | 44.0% | 41.0% / 13.0% | 69.0% |
| IdeaReward-Llama | 68.57% | 66.24% / 42.24% | 72.06% | 62.26% / 39.97% | 91.50% |
| IdeaReward-Qwen | 69.17% | 66.44% / 42.59% | 72.89% | 62.79% / 41.81% | 94.00% |
Across the review and impact-grounded splits, all three heuristic baselines remain close to random performance. On the annotation-grounded split, dataset and baseline mentions are somewhat predictive at 63.0% and 69.0%, but these heuristic baselines still remain far below IdeaReward-Llama and IdeaReward-Qwen, which achieve 91.5% and 94.0%, respectively. These results provide strong evidence that the outstanding performance of IdeaReward cannot be explained by exploiting superficial shortcuts and instead reflects actual learned judgements in idea quality.
W2 & Q1 (Continued)
(c) Authorship-reversal challenge. To directly test whether IdeaReward simply learns idea authorship detection (i.e. always preferring a human idea over an LLM idea), we construct an additional challenge set of 60 manually verified pairs. We select weak human ideas from zero-citation papers or strong-reject submissions, generate LLM ideas for the same research questions, and manually verify that the LLM idea is stronger. The preferred idea is therefore always LLM-generated, completely reversing the usual human-authorship preference.
| Evaluator | Accuracy |
|---|---|
| Always prefer human | 0.0% |
| IdeaReward-Llama | 71.6% |
| IdeaReward-Qwen | 76.7% |
Both models achieve over 70% accuracy despite this authorship-preference reversal, providing direct evidence that their predictions are not reducible to authorship detection and instead reflect idea-quality judgments.
We have incorporated these expanded analyses and appropriately qualified conclusions into the revised paper.
Q2: Best-of-N experiment size and lack of idea implementation
(1) Regarding the limited size of the BoN experiment.
We have expanded the BoN experiment from 30 to 60 topics and introduced DeepSeek-v4-pro as a second idea generator. We also added GLM-5.2 as an additional baseline idea selector. The expanded results are shown below:
| Baseline selector | Win | Loss | Tie | win-rate | 95% CI | |
|---|---|---|---|---|---|---|
| InternLM2-20B-Reward | 31 | 7 | 22 | 0.816 | 0.0001 | [0.684, 0.921] |
| GPT-5.5 | 19 | 11 | 30 | 0.633 | 0.20 | [0.467, 0.800] |
| GLM-5.2 (new) | 22 | 10 | 28 | 0.688 | 0.050 | [0.531, 0.844] |
In this expanded study, IdeaReward shows a clear advantage over all three baselines in test-time best-of-N idea selection, achieving non-tie win rates of 81.6% against InternLM2-20B-Reward, 63.3% against GPT-5.5, and 68.8% against GLM-5.2.
(2) Regarding the lack of idea implementation.
The objective of our work is to reliably evaluate idea quality before implementation, identifying promising idea proposals before costly execution. For readers concerned with executed project outcomes, the 3,085 review and impact-grounded instances evaluated in our main experiment are indeed derived from completed papers, directly linking idea quality to realized project-level outcomes. Therefore, IdeaReward’s strong performance on these 3,085 instances directly tests whether it can identify, from the idea alone, which idea proposals are more likely to develop into stronger projects. Large-scale implementation of new research ideas and execution-level validation remains beyond the scope of this work.
We sincerely thank the reviewer for the thoughtful review and for recognizing the paper’s presentation, the contribution of IdeaRewardBench, and our extensive experiments and analyses. Below, we address each concern in detail, with the corresponding revisions incorporated into the updated paper PDF.
W1: Positioning relative to LDC
We thank the reviewer for identifying this important missing related work. LDC (https://arxiv.org/pdf/2412.14626) is a closely related and complementary predecessor. While relevant, our work differs fundamentally from LDC in both contribution and methodology. Specifically:
Contribution. The primary objective of LDC is to solely improve research idea generation. In contrast, the primary objective of our work is to both diagnose and improve research idea evaluation. We believe that studying research idea evaluation is becoming increasingly important as LLMs become capable of generating ideas at scale, while idea evaluation remains the critical bottleneck. To the best of our knowledge, IdeaRewardBench is the first living benchmark for research idea evaluation, providing a unified testbed for scalar reward models, LLM judges, and specialized idea evaluators. On IdeaRewardBench, we extensively evaluate 23 evaluator baselines and identify systematic limitations of existing evaluators, in which generation-targeted work like LDC do not examine. For example, our main results show that even strong proprietary models like GPT-5.5 achieve near random performance at evaluating review and impact-grounded ideas, showing substantial room for improvement in idea evaluation. Our novelty-attack analysis further reveals that existing idea evaluators frequently misjudge already published ideas as novel, exposing a critical barrier to reliable deployment. Together, this comprehensive benchmark and its diagnostic findings constitute our central contribution. Our second contribution, IdeaReward, is a natural extension of the same scalable data-construction pipeline toward improving idea evaluation.
Methodology. Our reward-model training methodology also differs substantially from LDC. LDC employs multi-dimensional supervision for novelty, feasibility, and effectiveness, but derives all these signals from a single review-grounded source of conference peer reviews. In contrast, IdeaReward adopts source-level multi-signal supervision, combining review-grounded, impact-grounded, and synthetic idea preferences as training data. This source diversity is central to our methodology because any single source can introduce noise and biases. Combining multiple sources helps mitigate these source-specific biases by drawing on complementary evidence: review-grounded data provides ex-ante peer assessments at submission, impact-grounded data captures broader ex-post community influence over time, and controlled synthetic data provides targeted quality contrasts. These complementary signals reduce dependence on any single imperfect proxy. Consistent with this motivation, our main results in Section 4.1 and Table 1 show that single-source reward models generalize poorly to idea preferences grounded in other signals. At the dimension level, our dimension-specific degradation strategy teaches IdeaReward to learn dimensional-aware idea quality judgements, with results in Section 4.2 and Figure 3 showing that IdeaReward exhibits strong judgments across multiple dimensions including novelty, feasibility, and significance.
We have added LDC to our paper's related work section and revised our priority claims accordingly.