acceptodds
Under review as a conference paper at ICLR 2027

IdeaReward: Benchmarking and Training Reward Models for Research Idea Generation

Abstract

Research idea generation aims to produce research ideas that can be developed into publication-worthy research projects. However, as large language model (LLM)-based systems generate research ideas at scale, the central bottleneck shifts from producing candidate ideas to reliably evaluating, selecting, and improving them before costly research execution. Reward models offer a promising solution for this goal by learning to assign quality-aware reward scores from preference data. In this paper, we present a systematic study of reward modeling for research idea generation. We present IdeaRewardBench, the first multi-signal, living benchmark for research idea evaluation. IdeaRewardBench contains 3,185 idea preference instances grounded in complementary signals of idea quality, and is updated semiannually to reflect the latest research frontier. We evaluate a comprehensive set of 23 baselines on IdeaRewardBench, and find that all baselines struggle to distinguish high-quality research ideas: the best baseline achieves only 62.8% pairwise accuracy on the review-grounded split and 60.9% on the impact-grounded split, where 50% is the random baseline. We then present IdeaReward, the first reward model for research idea generation, trained on over 20,000 high-quality idea preference pairs constructed by a fully automated pipeline. IdeaReward establishes state-of-the-art performance on IdeaRewardBench, outperforming baselines based on much larger LLMs. We further show that IdeaReward improves downstream test-time best-of-N idea selection. We will release our data, models, and code upon publication.

Primary area:
datasets and benchmarks
Keywords:
Idea Generation, Idea Evaluation, Reward Models, AI for Research, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mLfdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents IDEAREWARDBENCH, which comprises 3,185 idea preference instances grounded in multiple complementary signals of idea quality. Evaluating 23 baselines on IDEAREWARDBENCH reveals substantial limitations in existing idea evaluators, highlighting the benchmark’s difficulty and diagnostic value. IDEAREWARD is the first reward model for research idea generation. IDEAREWARD is trained with a fully automated and scalable data construction pipeline, and achieves state-of-the-art performance on IDEAREWARDBENCH.

Critical strengths

This paper has a good struture and can be understandable. present a living benchmark for research idea evaluation and conduct extensive experiments and analyses on IDEAREWARD and evaluated baselines

Critical weaknesses
  1. Prior work, such as LDC: Learning to Generate Research Idea with Dynamic Control, has already explored training research idea reward models using peer-review feedback, including separate models for novelty, feasibility, and effectiveness. The authors should better clarify what is new beyond the multi-source preference data construction. This related work is also missing from the paper.
  2. The benchmark uses paper acceptance decisions and citation counts as proxies for idea quality. However, paper acceptance depends heavily on execution and presentation, while citations are influenced by popularity and research trends. It is unclear how reliably these signals reflect the quality of the original research ideas.
Questions for authors

The training and benchmark data are constructed using very similar signals, particularly review scores and citation counts. Although the papers come from different time periods, the model may mainly learn patterns associated with these proxies rather than generalizable research idea quality. More evaluation on independently annotated ideas would strengthen the claims.

The best-of-N experiment only covers 30 topics with seven candidate ideas each. The improvement over GPT-5.5 is also modest (9 wins versus 6 losses, with 15 ties). More importantly, the selected ideas are not actually implemented, making it difficult to tell whether better reward scores translate into more successful research projects.

5 comments

We sincerely thank the reviewer for the thoughtful review and for recognizing the paper’s presentation, the contribution of IdeaRewardBench, and our extensive experiments and analyses. Below, we address each concern in detail, with the corresponding revisions incorporated into the updated paper PDF.

W1: Positioning relative to LDC

We thank the reviewer for identifying this important missing related work. LDC (https://arxiv.org/pdf/2412.14626) is a closely related and complementary predecessor. While relevant, our work differs fundamentally from LDC in both contribution and methodology. Specifically:

  1. Contribution. The primary objective of LDC is to solely improve research idea generation. In contrast, the primary objective of our work is to both diagnose and improve research idea evaluation. We believe that studying research idea evaluation is becoming increasingly important as LLMs become capable of generating ideas at scale, while idea evaluation remains the critical bottleneck. To the best of our knowledge, IdeaRewardBench is the first living benchmark for research idea evaluation, providing a unified testbed for scalar reward models, LLM judges, and specialized idea evaluators. On IdeaRewardBench, we extensively evaluate 23 evaluator baselines and identify systematic limitations of existing evaluators, in which generation-targeted work like LDC do not examine. For example, our main results show that even strong proprietary models like GPT-5.5 achieve near random performance at evaluating review and impact-grounded ideas, showing substantial room for improvement in idea evaluation. Our novelty-attack analysis further reveals that existing idea evaluators frequently misjudge already published ideas as novel, exposing a critical barrier to reliable deployment. Together, this comprehensive benchmark and its diagnostic findings constitute our central contribution. Our second contribution, IdeaReward, is a natural extension of the same scalable data-construction pipeline toward improving idea evaluation.

  2. Methodology. Our reward-model training methodology also differs substantially from LDC. LDC employs multi-dimensional supervision for novelty, feasibility, and effectiveness, but derives all these signals from a single review-grounded source of conference peer reviews. In contrast, IdeaReward adopts source-level multi-signal supervision, combining review-grounded, impact-grounded, and synthetic idea preferences as training data. This source diversity is central to our methodology because any single source can introduce noise and biases. Combining multiple sources helps mitigate these source-specific biases by drawing on complementary evidence: review-grounded data provides ex-ante peer assessments at submission, impact-grounded data captures broader ex-post community influence over time, and controlled synthetic data provides targeted quality contrasts. These complementary signals reduce dependence on any single imperfect proxy. Consistent with this motivation, our main results in Section 4.1 and Table 1 show that single-source reward models generalize poorly to idea preferences grounded in other signals. At the dimension level, our dimension-specific degradation strategy teaches IdeaReward to learn dimensional-aware idea quality judgements, with results in Section 4.2 and Figure 3 showing that IdeaReward exhibits strong judgments across multiple dimensions including novelty, feasibility, and significance.

We have added LDC to our paper's related work section and revised our priority claims accordingly.

W2 & Q1: Do paper-level proxies reliably reflect idea quality, and does IdeaReward learn generalizable idea quality judgements?

We agree that "paper acceptance depends on execution and presentation, while citations are influenced by popularity and research trends". This is exactly why we combine multi-source signals rather than treating any source as a perfect label, which has been explained in our response to W1. We now respond to the reviewer's two concerns in detail.

(1) Do paper-level preferences reliably reflect idea quality?

To better answer this question, we expanded our pilot study of idea-paper preference agreement from 40 pairs to 200 benchmark instances, comprising 50 instances from each of four subsets: review-grounded pairs, review-grounded quadruples, impact-grounded pairs, and impact-grounded quadruples. Agreement between the human idea-level preferences and the gold paper-level preferences is shown below:

SplitNNAgreement between idea-level and paper-level preferences
Review-grounded pairs5088.0%
Review-grounded quadruples5084.0% Top-1 Acc / 90.3% PRA
Impact-grounded pairs5084.0%
Impact-grounded quadruples5082.0% Top-1 Acc / 92.0% PRA
Overall20084.5% top-choice agreement

The results show high agreement overall and across all four subsets, supporting paper-level preferences as informative and scalable proxies for comparative idea quality.

Moreover, deriving idea-quality signals from paper-level outcomes has clear methodological precedence in existing work, including LDC (https://arxiv.org/pdf/2412.14626, EMNLP 2026), as highlighted by the reviewer, and InnoEval (https://arxiv.org/pdf/2602.14367, ICML 2026). This approach is well motivated as the definition of an idea card is essentially the blueprint of a paper: a proposal-stage summary of its motivations, method, and experimental plan. Because a strong underlying idea is a key prerequisite for a strong paper, paper-level outcomes provide an informative and scalable, though inevitably imperfect, proxy for idea quality.

(2) Does IdeaReward learn generalizable idea-quality judgments?

(a) Generalization to independently annotated ideas. Our main results in Section 4.1 and Table 1 show that IdeaReward is the only approach that consistently achieves state-of-the-art performance across all three benchmark splits. In particular, its performance on the annotation-grounded split is especially informative because IdeaReward’s training data contains no human-annotated preferences. This split is therefore out of distribution with respect to the source of preference supervision. Despite receiving no human-annotated training preferences, IdeaReward-Llama and IdeaReward-Qwen achieve 91.5% and 94.0% overall accuracy on this split. On the human-vs-LLM subset, IdeaReward outperforms the second-best evaluator by 15 percentage points. These results provide direct evidence that IdeaReward learns idea-quality judgments that generalize beyond the review- and impact-grounded distributions constituting the majority of its training data.

(b) Ruling out superficial shortcuts. In this analysis, we further show that IdeaReward learns genuine idea quality judgement rather than superficial shortcuts. We compare IdeaReward against three naive heuristic baselines based on idea card length, the card's number of dataset mentions, and the card's number of baseline mentions. The card-length baseline is evaluated on the complete benchmark. The two mention-count baselines are evaluated on 300 instances comprising 50 pairs and 50 quadruples from each review and impact-grounded split, together with all 100 annotation-grounded pairs.

EvaluatorReview pair acc.Review quadruple PRA / top-1Impact pair acc.Impact quadruple PRA / top-1Annotation pair acc.
Card length50.1%52.0% / 27.2%48.7%47.4% / 24.9%55.0%
Dataset mentions51.0%56.7% / 29.0%54.0%53.8% / 35.0%63.0%
Baseline mentions57.0%53.8% / 33.8%44.0%41.0% / 13.0%69.0%
IdeaReward-Llama68.57%66.24% / 42.24%72.06%62.26% / 39.97%91.50%
IdeaReward-Qwen69.17%66.44% / 42.59%72.89%62.79% / 41.81%94.00%

Across the review and impact-grounded splits, all three heuristic baselines remain close to random performance. On the annotation-grounded split, dataset and baseline mentions are somewhat predictive at 63.0% and 69.0%, but these heuristic baselines still remain far below IdeaReward-Llama and IdeaReward-Qwen, which achieve 91.5% and 94.0%, respectively. These results provide strong evidence that the outstanding performance of IdeaReward cannot be explained by exploiting superficial shortcuts and instead reflects actual learned judgements in idea quality.

W2 & Q1 (Continued)

(c) Authorship-reversal challenge. To directly test whether IdeaReward simply learns idea authorship detection (i.e. always preferring a human idea over an LLM idea), we construct an additional challenge set of 60 manually verified pairs. We select weak human ideas from zero-citation papers or strong-reject submissions, generate LLM ideas for the same research questions, and manually verify that the LLM idea is stronger. The preferred idea is therefore always LLM-generated, completely reversing the usual human-authorship preference.

EvaluatorAccuracy
Always prefer human0.0%
IdeaReward-Llama71.6%
IdeaReward-Qwen76.7%

Both models achieve over 70% accuracy despite this authorship-preference reversal, providing direct evidence that their predictions are not reducible to authorship detection and instead reflect idea-quality judgments.

We have incorporated these expanded analyses and appropriately qualified conclusions into the revised paper.

Q2: Best-of-N experiment size and lack of idea implementation

(1) Regarding the limited size of the BoN experiment.

We have expanded the BoN experiment from 30 to 60 topics and introduced DeepSeek-v4-pro as a second idea generator. We also added GLM-5.2 as an additional baseline idea selector. The expanded results are shown below:

Baseline selectorWinLossTiewin-ratepp95% CI
InternLM2-20B-Reward317220.8160.0001[0.684, 0.921]
GPT-5.51911300.6330.20[0.467, 0.800]
GLM-5.2 (new)2210280.6880.050[0.531, 0.844]

In this expanded study, IdeaReward shows a clear advantage over all three baselines in test-time best-of-N idea selection, achieving non-tie win rates of 81.6% against InternLM2-20B-Reward, 63.3% against GPT-5.5, and 68.8% against GLM-5.2.

(2) Regarding the lack of idea implementation.

The objective of our work is to reliably evaluate idea quality before implementation, identifying promising idea proposals before costly execution. For readers concerned with executed project outcomes, the 3,085 review and impact-grounded instances evaluated in our main experiment are indeed derived from completed papers, directly linking idea quality to realized project-level outcomes. Therefore, IdeaReward’s strong performance on these 3,085 instances directly tests whether it can identify, from the idea alone, which idea proposals are more likely to develop into stronger projects. Large-scale implementation of new research ideas and execution-level validation remains beyond the scope of this work.

Reviewer_VigEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a benchmark for research idea evaluation, as well as a new reward model for assessing research idea quality. On the benchmark, the authors find that existing baselines struggle to distinguish between high-quality research ideas. Motivated by this problem, the authors introduce a new reward model for research idea generation that obtains SOTA performance on the benchmark. Overall, the paper is well written and easy to follow.

Critical strengths

Addresses a timely problem in a methodologically sound way. Decomposing idea rewards in a multi-horizon view (review-grounded, impact-grounded, and annotation-grounded) is a clever idea that considers the many time horizons along which scientific work is rewarded and judged. The findings that (1) existing baselines perform poorly on IdeaRewardBench and (2) general reward models fail to capture research idea quality both substantiate the problem that frontier LLMs are susceptible to reward-hacking in modern post-training environments.

Informative follow-up analysis. This work addresses an interesting range of questions in its analysis, including whether idea evaluators can separate research ideas as well as humans can, which dimensions of idea quality idea evaluators can capture, and whether idea evaluators can actually recognize unoriginal ideas. The failure rates on the “novelty attack” indicate meaningful room to improve retrieval-equipped idea evaluators, as the authors mention. This is an important finding that now looms as AI-assisted research now dominates modern peer review.

Critical weaknesses

Limited to the AI/ML domain The paper focuses predominantly on AI and machine learning rewards and data (e.g. NeurIPS, ICLR, ICML for the reviews, arXiv cs.AI, cs.CL, cs.LG for the impact, and ICLR submissions for the annotations). For rebuttals, the authors can consider explicitly delineating the scope of rewards they are considering here, and including commentary about the expected generality (or lack thereof) of the framework to other scientific fields.

For the impact-grounded rewards, not all citations are equal. See [1] for recent work in the science of science that distinguishes between foundational, extensional, and generalizational citations. Additionally, some citations that point out weaknesses in existing papers can actually be a negative signal for future impact/reward, rather than positive. For rebuttals, the authors can consider decomposing citations into various categories, and using this to ground citation impact in a more robust and accurate way.

[1] https://arxiv.org/abs/2510.03240

Questions for authors

Could you increase the font size of Figure 3, 4, and 5 so they are readable? Generally it’s good practice to keep figure font size at least as big as the main body font.

4 comments

We sincerely thank the reviewer for the comprehensive and thoughtful review, and for recognizing the methodological soundness of our work and the breadth and insights of our analyses. We especially appreciate the positive assessment of our novelty-attack analysis, which constitutes an important contribution beyond our benchmark and reward-model training recipe. Below, we address the reviewer’s concerns in detail.

W1: Scope beyond the AI/ML domain

We thank the reviewer for highlighting this important scope consideration. Our work introduces a general framework for constructing living idea-evaluation benchmarks and training specialized idea reward models, with its empirical validation currently focused on the AI/ML domain. We expect the framework to transfer to other scientific fields where sufficiently rich peer-review and citation data are available. Such extensions may also require adapting the idea-card schema to each discipline’s research practices.

Our scope framing follows prior research that proposes generalizable methods to scientific discovery while initially validating them in the AI/ML domain, such as The AI Scientist (https://arxiv.org/pdf/2606.15497) (Nature 2026) and Can LLMs Generate Novel Research Ideas? (https://arxiv.org/pdf/2409.04109) (ICLR 2025). We have revised the Limitations section of the paper to clarify the current AI/ML scope and discuss the framework’s expected generalizability.

W2: Citation signals are imperfect and can be further decomposed

We agree that raw citation counts provide an imperfect measure of scientific impact. We thank the reviewer for pointing us to the work of Fang and Evans, which distinguishes foundational, extensional, and generalizational citations through downstream citation-network structures. Incorporating these distinctions would be a valuable addition to our impact-grounded data. However, unfortunately, we found that the work of Fang and Evans does not currently provide a publicly accessible implementation, and applying its method at our scale would require constructing a large-scale citation-network pipeline. This extension is beyond the scope of our current paper and cannot be reliably completed within the rebuttal period. We do acknowledge that this represents a valuable direction for further exploration, and will consider incorporating fine-grained citation-role and citation-context analysis in future work.

Although our current pipeline does not incorporate fine-grained citation-role analysis, we apply several safeguards to ensure the reliability of our constructed impact-grounded idea preferences. We normalize citation counts by publication-time, remove idea groups containing within-group citation edges, require citation gaps of at least 2:1 for pairs and 4:3:2:1 for quadruples, and require the highest-ranked paper to have at least 30 raw citations. These safeguards promote high-quality idea preference instances with clear separation between compared ideas.

We would also like to highlight the rationale behind our multi-signal design. We agree that any individual source of idea quality can introduce noise, including the potential limitations of citation data identified by the reviewer. This is precisely why multi-signal construction lies at the center of our approach. Rather than relying on any single signal, such as citation data, we combine multiple idea quality sources with distinct, complementary strengths and limitations. Review-grounded data provides well-rounded assessments of novelty, soundness, and significance, although these may reflect the views of a small reviewer group at paper submission time. Impact-grounded data captures broader community-level uptake and influence over longer time periods. Annotation-grounded preferences provide direct idea-level judgments independent of final paper outcomes. Together, these signals provide a more comprehensive and robust representation of idea quality.

Question: Figure readability

Thank you for your feedback. We have increased the font sizes in Figures 3, 4, and 5 in the updated PDF.

We sincerely thank the reviewer for their time and thoughtful feedback. We hope that we have addressed all questions and concerns, and would be happy to provide any further clarification or information.

Reviewer_VigEOpenReview ↗

Reviewer Response

Thanks to the authors for their rebuttal response. As of now, I maintain my current score, but wanted to share that the Fang and Evans code is available as of 2 weeks ago (https://github.com/fhbzc/Generalization-replication). I think engaging more rigorously with methods and techniques in the extensive science of science literature is crucial for this work to maximize its potential and be a strong accept.

Dear Reviewer VigE,

We sincerely thank you for taking the time to review our paper and respond to our rebuttal. We missed Fang and Evans’s code repository because the previously shared version of their paper (https://arxiv.org/abs/2510.03240) did not include a link, so we’re grateful you shared it with us. We will study their implementation and consider whether their method could further improve how we use citation data to construct IdeaRewardBench’s impact-grounded split.

Best regards, The Authors

Reviewer_9z1iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward modeling for research ideas and makes two contributions. The first is IdeaRewardBench, a benchmark of 3,185 preference instances. Labels for the review-grounded split come from acceptance decisions and reviewer scores at NeurIPS 2025, ICLR 2026, and ICML 2026. Labels for the impact-grounded split come from normalized citation counts of recent arXiv papers. A small annotation-grounded split has 100 pairs labeled by two annotators. Each paper is converted into an anonymized five-field idea card. The second contribution is IdeaReward, a reward model trained on 20523 automatically constructed pairs. IdeaReward reaches roughly 69% and 73%, and 96% on the human-vs-LLM subset.

Critical strengths
  1. A shared benchmark for idea evaluation is clearly needed, and combining review, citation, and annotation signals is more robust than relying on any one of them. The baseline coverage is broad: general RMs, LLM judges, and many specialized idea evaluators are run under a single protocol, with adaptations and costs documented.
Critical weaknesses
  1. The 3,085 review- and impact-grounded instances are labeled from acceptance decisions or citations. These signals judge the full paper, not the extracted idea card. Yet whether they carry over to idea cards is checked on only 40 pairs (Appendix A), and none of the 589 quadruples are checked. Without this, it’s unclear whether low baseline accuracy reflects weak evaluators or noisy labels. Please validate a much larger sample, including quadruples, and report a human upper bound on idea cards.
  2. Idea cards may carry shortcuts from the final paper. Cards are extracted from completed papers, so the experiment plan reflects experiments that were actually run. Accepted or highly cited papers tend to have more thorough experiments, which produces longer and more detailed cards. Simple baselines such as card length or the number of datasets and baselines mentioned would show how much of the task can be solved without judging the idea itself.
  3. The human-vs-LLM subset may measure source detection. Human ideas are extracted from full papers and LLM ideas are generated from seeds, so their style likely differs. The training set includes 826 synthetic pairs that, by construction, label the human idea as better than the LLM idea (Table 6), which directly teaches this shortcut. IdeaReward reaches 96.0 on this subset, compared with 8.0–41.0 for general RMs (Table 1), which looks more like source detection than quality judgment.
  4. Several experiments are small. The annotation split has 100 pairs, labeled by the same two annotators. The BoN study covers 30 topics, and against GPT-5.5 the result is 9 wins, 6 losses, and 15 ties.
Questions for authors
  1. Only reading the idea cards, how often do human annotators agree with the paper-level labels on a larger sample, and does the agreement differ between pairs and quadruples, or between the review and impact splits?
  2. What is the inter-annotator agreement (e.g., Cohen’s κ) on the annotation split? Is the BoN result against GPT-5.5 statistically significant, and does it hold with more topics or other generators?
  3. How well do simple heuristics such as card length or the number of datasets and baselines in the experiment plan perform on the benchmark? How does IdeaReward’s score correlate with card length?
5 comments

We sincerely thank the reviewer for the comprehensive and constructive review, and for recognizing the value of IdeaRewardBench, its multi-signal design, and the breadth of our baseline evaluation. The reviewer raises several insightful and important questions that help us further strengthen the paper. Below, we address each concern, with corresponding revisions incorporated into the updated paper PDF.

W1 & Q1: Do paper-level preferences reliably transfer to idea cards?

Upon the reviewer's request, we expanded our pilot study of idea-paper preference agreement from 40 pairs to 200 benchmark instances, comprising 50 instances from each of four IdeaRewardBench subsets: review-grounded pairs, review-grounded quadruples, impact-grounded pairs, and impact-grounded quadruples. Following the original blind protocol, two human annotators independently ranked the anonymized idea cards in each instance and resolved disagreements through discussion. Agreement between the final human idea-level preferences and the gold paper-level preferences is shown below:

SplitNNAgreement between final idea-level and paper-level preferences
Review-grounded pairs5088.0%
Review-grounded quadruples5084.0% Top-1 Acc / 90.3% PRA
Impact-grounded pairs5084.0%
Impact-grounded quadruples5082.0% Top-1 Acc / 92.0% PRA
Overall20084.5% top-choice agreement

The results show high agreement overall and across all four subsets, demonstrating that paper-level preferences provide reliable proxies for comparative idea quality, even in the more challenging quadruple setting.

To answer Q1, we also report inter-annotator agreement before disagreement resolution. For quadruples, agreement and Cohen’s κ\kappa are computed based on the annotators’ top-ranked ideas.

SplitNNInter-annotator agreementCohen's κ\kappa
Review-grounded pairs5086.0%0.700
Review-grounded quadruples5076.0%0.669
Impact-grounded pairs5090.0%0.798
Impact-grounded quadruples5070.0%0.597
Overall20080.5%0.710

Moreover, deriving idea-quality signals from paper-level outcomes has clear methodological precedence in existing work, including LDC (https://arxiv.org/pdf/2412.14626, EMNLP 2026) and InnoEval (https://arxiv.org/pdf/2602.14367, ICML 2026). This approach is well motivated as the definition of an idea card is essentially the blueprint of a paper: a proposal-stage summary of its motivations, method, and experimental plan. Because a strong underlying idea is a key prerequisite for a strong paper, paper-level outcomes provide an informative and scalable, though inevitably imperfect, proxy for idea quality.

We have updated Section 2.2 and Appendix A of the revised paper with this expanded study.

W2 & Q3: Do idea cards expose superficial shortcuts?

We thank the reviewer for raising this important question. Resistance to shortcut bias is essential for a reliable benchmark, and this analysis allows us to show that IdeaReward learns genuine idea-quality judgements rather than superficial shortcuts.

Following the reviewer’s suggestion, first, we compared our IdeaReward models on IdeaRewardBench against three simple heuristic baselines based on idea card length, dataset mentions, and baseline mentions. The card-length baseline directly ranks idea cards by word count, favoring longer cards, and is evaluated on the full IdeaRewardBench. For the dataset and baseline-mention baselines, we use GLM-5.2 to count the respective mentions and rank cards by these counts, favoring cards with more mentions. These two baselines are evaluated on a 300-instance subset of IdeaRewardBench, comprising 50 randomly sampled pairs and 50 quadruples from each of the review and impact-grounded splits, together with the full 100 annotation-grounded pairs.

EvaluatorReview pair acc.Review quadruple PRA / top-1Impact pair acc.Impact quadruple PRA / top-1Annotation pair acc.
Card length50.1%52.0% / 27.2%48.7%47.4% / 24.9%55.0%
Num Dataset mentions (300-instance subset)51.0%56.7% / 29.0%54.0%53.8% / 35.0%63.0%
Num Baseline mentions (300-instance subset)57.0%53.8% / 33.8%44.0%41.0% / 13.0%69.0%
IdeaReward-Llama68.57%66.24% / 42.24%72.06%62.26% / 39.97%91.50%
IdeaReward-Qwen69.17%66.44% / 42.59%72.89%62.79% / 41.81%94.00%

Across the review and impact-grounded splits, all three heuristic baselines remain close to random performance. On the annotation-grounded split, dataset and baseline mentions are somewhat predictive at 63.0% and 69.0%, respectively, reflecting a mild distributional bias: human ideas can indeed contain more thorough experimental plans than LLM-generated ideas. Nevertheless, these heuristic results still remain far below IdeaReward-Llama and IdeaReward-Qwen, which achieve 91.5% and 94.0%, respectively. These results show that the benchmark’s 3,085 review and impact-grounded instances contain no substantial shortcuts based on card length or mention counts, while the smaller annotation split retains a mild but realistic distributional signal. Both IdeaReward models significantly outperform every heuristic across all splits and metrics, providing strong evidence that their performance cannot be explained by exploiting these surface cues and instead reflects learned judgements in idea quality.

Second, we examine the relationship between idea card length and IdeaReward scores. We calculate the word count of all 7,548 idea cards in IdeaRewardBench, sort the cards by length, divide them into ten near-equal bins of 754 or 755 cards. We then compute the mean raw score assigned by each IdeaReward model within each bin.

Length decilennMean length (words)IdeaReward-Llama meanIdeaReward-Qwen mean
1 (shortest)755403.81.7691.978
2755455.12.0022.234
3755475.82.0012.232
4755492.32.0142.231
5755507.52.0252.231
6755522.91.9642.212
7755539.01.9812.227
8755557.22.0132.252
9754581.21.9392.215
10 (longest)754637.41.9662.268

Both models assign lower scores to the shortest decile, indicating a modest short-card effect. This is unsurprising because exceptionally short idea cards may provide less complete descriptions of the proposed method and experimental design. Across the remaining nine bins, however, mean scores remain nearly flat. Neither IdeaReward-Llama nor IdeaReward-Qwen shows a monotonic score increase with card length, indicating no systematic bias in favor of longer cards.

W3: Does the Human-vs-LLM split primarily measure source detection?

We thank the reviewer for raising this important concern. First, we clarify that the 826 fixed-label human-vs-LLM pairs are only one component of our automated data-construction pipeline and account for approximately 4% of IdeaReward’s 20,523 training pairs. The training set also contains 865 human-vs-LLM pairs labeled through pairwise quality evaluation by GPT-5.6-Sol rather than by a fixed preference for the human idea. Among these pairs, the LLM idea is preferred in 135 cases:

SourceHuman preferredLLM preferredTotal
Impact-grounded27690366
Review-grounded45445499
Total730135865

Thus, the training data include explicit counterexamples to an “always prefer human” rule and do not uniformly associate the human source with a positive label.

Furthermore, we directly test this potential shortcut using a source-reversal challenge set. We deliberately select 60 weak human ideas from papers with zero citations or strong-reject submissions. For each human idea, we generate LLM ideas from the same research question and manually verify that the selected LLM idea is stronger than the human idea. By construction, the gold-preferred idea is always generated by an LLM, meaning that an “always prefer human” baseline achieves 0% accuracy. Our results are shown below:

EvaluatorAccuracy
Always prefer human0.0%
IdeaReward-Llama71.6%
IdeaReward-Qwen76.7%

Both IdeaReward models achieve over 70% accuracy despite reversing the human-source preference tendency. These results provide direct evidence that the models assess idea quality rather than simply applying a learned preference for human-authored ideas. We have added this challenge-set analysis to Appendix F of the revised paper.

W4 & Q2: The annotation split and the best-of-N experiment

(1) Regarding the annotation split. The annotation-grounded split of IdeaRewardBench contains 100 instances, making it smaller than the 3,085 automatically constructed review and impact-grounded instances. However, it is carefully annotated by experts and designed for broad coverage. It includes both human-vs-LLM and LLM-vs-LLM comparisons, two idea generation settings based on either topic or research question paired with motivations, and six LLM idea generators spanning proprietary and open-source models of varying sizes (See Appendix D for more details). Together, these design choices make it a compact yet diverse and high-quality evaluation split. We thank the reviewer for this suggestion and will consider further expanding the annotation-grounded split for the camera-ready version. To answer Q2, Cohen’s κ\kappa between the two annotators before disagreement resolution is 0.84, indicating strong inter-annotator consistency.

(2) Expanding the best-of-N experiment We first assess the statistical significance of the original best-of-N experiment.
For all comparisons, we apply two-sided exact binomial sign tests to wins and losses after excluding ties, and report bootstrap 95% confidence intervals for the non-tie win rates. Against InternLM2-20B-Reward, IdeaReward's 14 wins, 4 losses, and 12 ties yield p=0.031p=0.031, with a 95% confidence interval of [0.571,0.947][0.571, 0.947], indicating a statistically significant advantage. Against GPT-5.5, IdeaReward's 9 wins, 6 losses, and 15 ties yield p=0.61p=0.61, with a 95% confidence interval of [0.333,0.846][0.333, 0.846]. We acknowledge that this original result is not statistically significant.

Following the reviewer’s recommendation, we expanded the experiment from 30 to 60 topics and introduced DeepSeek-v4-pro as a second idea generator. We also added GLM-5.2 as an additional baseline idea selector. For each topic, we compare IdeaReward’s selected idea against that of each baseline and record a win, loss, or tie. The expanded results are shown below:

Baseline selectorWinLossTiewin-ratepp95% CI
InternLM2-20B-Reward317220.8160.0001[0.684, 0.921]
GPT-5.51911300.6330.20[0.467, 0.800]
GLM-5.2 (new)2210280.6880.050[0.531, 0.844]

In the expanded study, IdeaReward significantly outperforms InternLM2-20B-Reward and shows an advantage over GLM-5.2 at the boundary of statistical significance. Against GPT-5.5, IdeaReward achieves a favorable 63.3% non-tie win rate, although the difference remains statistically non-significant. Nevertheless, this directional advantage is still encouraging because IdeaReward uses an 8B open-weight backbone, whereas GPT-5.5 is a large, proprietary model. We have incorporated the expanded experiment, significance analysis, and appropriately qualified conclusions into the revised paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Paper Revisions and Key Contributions

Dear Reviewers,

We would like to express our sincere gratitude to all reviewers for their dedicated time and valuable comments. Following the reviewers’ feedback, we have provided further evidence on two key questions: (1) How well do paper-level signals from peer reviews and citations reflect the quality of the underlying ideas? (2) Does IdeaReward learn to judge actual idea quality rather than simple shortcuts such as idea card length?

We have updated the paper PDF to carefully address both questions. Key updates to our revised paper include:

  1. Expanded our pilot study on idea–paper preference agreement. We expanded our human pilot study of idea–paper preference agreement from 40 pairs to 200 instances and found that final human judgments of the idea cards matched the paper-level preferences in 84.5% of instances. This further supports our use of review and citation-based paper outcome preferences as informative and scalable proxies for comparative idea quality. We have updated Section 2.2 and Appendix A of our paper to report the expanded study.

  2. Conducted additional analyses on superficial shortcuts. We further added three analysis experiments to test superficial shortcuts. (1) Comparing simple heuristic baselines: We find that ranking ideas by card length, number of dataset mentions, or number of baseline mentions does not reliably predict preferences across IdeaRewardBench. Furthermore, across all splits, IdeaReward significantly outperforms all heuristic baselines by 6.8 to 25.0 percentage points, providing evidence that it learns to judge idea quality beyond these superficial shortcuts. (2) Analyzing IdeaReward scores by card length: We grouped all 7,548 idea cards into ten bins by word count and calculated each IdeaReward model’s mean raw score for each bin. IdeaReward scores did not consistently increase with idea length, showing that IdeaReward does not simply favor longer ideas. (3) Idea authorship-reversal test: On 60 manually verified pairs in which an LLM-generated idea was stronger than a human idea, both IdeaReward models successfully selected the stronger idea in over 70% of cases, showing that IdeaReward does not simply converge to human authorship detection. Taken together, these three analyses consolidate evidence that IdeaRewardBench reliably measures differences in idea quality and that IdeaReward learns to recognize those differences rather than relying on superficial shortcuts. The corresponding results are updated in Appendix F of our paper.

  3. Expanded our BoN application experiment. We doubled the best-of-N experiment from 30 to 60 research topics, added a second idea generator, and included GLM-5.2 as an additional idea selector. Excluding ties, ideas selected by IdeaReward were preferred to those selected by InternLM2-20B-Reward, GPT-5.5, and GLM-5.2 in 81.6%, 63.3%, and 68.8% of cases. The corresponding results are updated in Section 5 of our paper.

As a recap, our paper's core contributions are outlined below:

  1. We present IdeaRewardBench, the first living benchmark for research idea evaluation, with 3,185 high-quality idea preference instances covering three diverse sources. As LLMs generate ideas at scale, the bottleneck shifts from generating idea candidates to evaluating which are worth pursuing, and IdeaRewardBench provides the first unified testbed for scalar reward models, LLM judges, and specialized idea evaluators. Rather than relying on a single source of idea quality, IdeaRewardBench draws on three complementary idea quality signals: peer reviews, citation impact, and direct expert annotation, reducing reliance on the biases of any single source. We update the benchmark semiannually to ensure that it continues to reflect the latest research frontier.

  2. Using IdeaRewardBench, we present an extensive, systematic analysis of where current idea evaluators fall short. We evaluate 23 baselines across three categories and find that the best pairwise accuracies reach only 62.8% on review-grounded pairs and 60.9% on impact-grounded pairs, compared with 50% for random guessing. Our analyses show several key insights on the failure modes of current idea evaluators. (1) Judging idea feasibility is a particular weakness. (2) Although most evaluators can identify clear rejects, they struggle to distinguish ideas of higher quality, including weak rejects, weak accepts, and strong accepts. (3) Our novelty-attack analysis reveals a further limitation: even idea evaluators that can retrieve prior work frequently misjudge already published ideas as novel. These results highlight specific weaknesses that future idea evaluation methods need to address.

  3. We present IdeaReward, a dedicated reward model for research idea generation. Our baseline evaluation shows that models trained on a single quality signal generalize poorly to preferences grounded in other signals. To address this limitation, we train IdeaReward on 20,523 preference pairs constructed by a fully automated pipeline that combines diverse review-grounded, impact-grounded, and controlled synthetic idea preferences. IdeaReward sets new state of the art results on all splits of IdeaRewardBench, significantly outperforming idea evaluators based on much larger models such as GPT-5.5. These gains also extend to test-time best-of-N idea selection: across 60 research topics, ideas selected by IdeaReward outperform those selected by GPT-5.5, GLM-5.2, and InternLM2-20B-Reward.

We will release the benchmark, training data, models, and code upon publication. We sincerely thank the reviewers again for helping us further strengthen the paper. If any questions or concerns remain, we would be glad to address them.

Warm regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.