acceptodds
Under review as a conference paper at ICLR 2027

AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment

Abstract

Research ideas are always proposed under resource constraints: a plan must fit the available compute, API credit, and time. Yet evaluations of research-ideation agents mostly score unconstrained novelty or overlap with a published paper, and their LLM judges are rarely checked against peer review. We introduce AlphaIdeaBench, a benchmark of 100 ideation tasks (50 workshop calls for papers and 50 open problems stated in surveys), each paired with an explicit resource budget and a shared proposal schema. An LLM judge scores each proposal on 12 dimensions covering validity, novelty, and feasibility. To validate the judge, we compare its scores on four novelty axes (problem, method, scenario, and insight) with scores derived from reviewer comments on 600 papers from ICLR 2026, NeurIPS 2025, and ICML 2025. For problem and method novelty, Gemini 3.7 Flash is within one point of the reviewer-derived score on 94–97% of papers; agreement on scenario novelty is relatively lower. We evaluate eight published harnesses on a shared backbone and, within one harness, eight backbone LLMs. CoI-Agent obtains the highest overall score, narrowly ahead of SciMON, which leads on method and insight novelty. All harnesses score low on scenario novelty and rarely change the problem setting. The backbone largely determines which open problem a proposal targets, whereas the harness determines whether a proposal addresses one problem or combines several. Weaker backbones such as Llama 4 Scout fail to carry out that harness's strategy and score low on budget feasibility. Finally, when only the budget is varied on three topics, a larger budget changes the experimental setting on compute-intensive topics but only adds tests on an inexpensive one. We release the tasks, rubrics, and scores to foster standardized, reproducible benchmarking.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, AI Agents, Research Idea Generation, Automated Research, Benchmarking, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kLYYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AlphaIdeaBench, a benchmark for evaluating the automated research idea generation capability. It does not focus on whether the agent can produce "paper-like" texts, but rather: under specific constraints of computing power, budget, time, and human input, can it propose scientifically valid, novel, and feasible research plans.

Critical strengths
  1. Explicit resource constraints make the evaluation practically relevant. Research ideas must be feasible within available compute, time, funding, and data constraints, rather than merely novel on paper. AlphaIdeaBench explicitly incorporates these constraints and assesses whether the full experimental plan—including baselines, ablations, and repeated runs—fits the stated resource budget. This makes the evaluation more practically relevant than approaches that assess ideation without accounting for resource limitations.

  2. The fine-grained rubric distinguishes different forms of scientific contribution. The twelve-dimensional rubric separates validity, novelty, and feasibility. In particular, decomposing novelty into problem, method, scenario, and insight novelty helps distinguish methodological improvements within an unchanged research setting from contributions that introduce new questions, scenarios, or mechanistic insights. The publicly available scoring descriptors and checklists also make the evaluation protocol easier to reuse.

Critical weaknesses
  1. The use of real reviewer comments provides a meaningful human reference. However, Gemini both converts these comments into axis-specific reference scores and serves as the judge being evaluated. The reliability of this conversion is insufficiently established, leaving uncertainty about how much the reported agreement reflects human judgments rather than shared model biases.
  2. The budget study does not adequately distinguish budget effects from generation variability. It covers only three topics, with one proposal per topic–budget condition and no reported within-condition repeats. Appendix E.3 illustrates the concern: rerunning the tool-use topic under the same 200 GPU-hour budget produces a different proposal, with scenario novelty dropping from 4 to 2. Thus, the examples demonstrate possible changes across budgets but do not establish that these are systematic responses to resource constraints rather than run-to-run variation. This limits confidence in a central empirical finding; repeated generations and uncertainty estimates are needed to substantiate it.
Questions for authors

1.How was the mapping from reviewer comments to the four novelty scores validated to ensure that it faithfully reflects reviewer judgments rather than the mapping model’s own interpretations? 2. Were any repeated generations conducted for the same topic–budget condition beyond the rerun described in Appendix E.3? If so, how does within-condition variability compare with the differences across budgets? This would help determine whether the reported changes reflect systematic budget responses rather than run-to-run variation. 3. If the model being evaluated is replaced with a more powerful one, such as GPT-6 or Opus-5, will their scores on the list be significantly higher than those of the current weaker models being tested? Which aspect will be stronger?

8 comments

Thank you for the constructive review and for recognizing the practical value of explicit resource constraints and the diagnostic role of the fine-grained rubric. Following the review, we conducted additional human-validation and repeated-generation experiments to directly examine the two main empirical concerns. We also clarify the implications of our existing backbone comparison for stronger future models.

1. Faithfulness of reviewer-comment mapping

The review asks whether the four novelty reference scores extracted from human reviewer comments faithfully reflect the reviewers’ judgments rather than Gemini’s own interpretation. We added two independent human checks.

(a) Independent human scoring of reviewer comments.
We sampled 30 papers, with 10 each from ICLR, ICML, and NeurIPS. Three experts independently read the original reviewer comments and assigned scores on the same four novelty dimensions. The annotators were blind to all AI-generated scores.

ExpertWithin ±1Exact
199.17%82.5%
2100%87.5%
399.17%83.3%

The exact agreement between the AI-derived reference scores and the independent human scores is 82.5–87.5%, while essentially all judgments are within one point. This provides a direct human validation of the reviewer-comment-to-score mapping rather than relying only on Gemini self-consistency.

(b) Direct audit of score/rationale faithfulness.
We further sampled another 60 papers, with 20 from each venue, and asked three experts to directly judge whether the generated score and its accompanying rationale faithfully represented the original reviewer comments. The corresponding faithfulness rates were 93.3%, 94.6%, and 93.8%.

As complementary evidence, we embedded each sentence of the original reviews using gemini-embedding-001 and matched each generated rationale to its most semantically similar source sentence. The mean maximum similarity is:

DatasetSimilarity
Overall0.874
ICLR0.875
ICML0.874
NeurIPS0.873

We treat the expert audit as the primary evidence, while the semantic-similarity analysis provides an additional check that the generated rationales are grounded in the original review text.

Takeaway.
These two studies independently validate both the assigned novelty scores and the rationales used to derive them, substantially reducing the concern that the reviewer-derived reference scores primarily reflect the mapping model’s own preferences.


2. Budget effects versus generation variability

The review correctly identifies that the original budget study used one proposal per topic–budget condition, making it difficult to separate systematic budget effects from generation variability. Following this suggestion, we repeated every condition.

(a) Clarifying the 4→2 tool-use case.
We first clarify the two tool-use proposals generated under the same Academic (200 GPU-hour) condition. AsyncTx-Agent received a Scenario Novelty score of 4 because it introduced an asynchronous operating setting involving “cross-task contamination with out-of-order network latency perturbations, sibling-task hijacking, and speculative state corruption.” The judge further noted that these constraints “necessitate targeted architectural redesign,” including transactional isolation, two-phase staging, and branch-level speculative rollback.

In contrast, the same-budget rerun, TaintTool-Graph, received 2 because its TaintTool-Bench setting “builds directly on top of existing InjecAgent and BIPIA attack taxonomies,” and the judge characterized it as “the standard academic benchmark setup for evaluating indirect prompt injection defenses and schema-compliant tool calling.”

Thus, the score difference reflects substantively different scenarios generated under the same resource condition, confirming the reviewer’s concern that run-to-run generation variability can matter.

(b) Repeated-generation results.
To examine this variability more systematically, we conducted three additional full CoI-Agent generations for each of the 3 topics × 3 budget conditions, yielding 27 additional proposals. The table below reports statistics for these additional runs only; the original proposals and the previously reported tool-use rerun are reported separately so that the new repeated-generation experiment is clearly identified.

TopicLowMediumHigh
Tool use2.00 (0.00)1.33 (0.33)1.00 (0.00)
VLA1.33 (0.33)2.33 (0.33)3.00 (0.00)
Interpretability1.00 (0.00)3.33 (0.33)4.00 (0.00)

Values are mean (sample variance) over three additional generations.

Among the additional runs, the within-condition sample variance is either 0 or 0.33.

(c) Topic-dependent budget responses.
The repeated runs show different patterns across topics. For VLA and interpretability, the mean Scenario Novelty scores increase monotonically across the three resource levels:

  • VLA: 1.33 → 2.33 → 3.00.
  • Interpretability: 1.00 → 3.33 → 4.00.

The corresponding low-to-high differences are 1.67 and 3.00 points. In contrast, tool use shows no positive trend (2.00 → 1.33 → 1.00). Moreover, the original score-4 AsyncTx-Agent was not reproduced in the subsequent generations, indicating that an individual high-scoring proposal should not be interpreted as a systematic budget effect.

Takeaway.
The repeated-generation analysis therefore refines our original conclusion. We do not claim that increasing the resource budget always increases Scenario Novelty. Instead, the observed response to the resource envelope is topic-dependent: the repeated-run mean increases monotonically for VLA and interpretability, but not for tool use, while non-negligible run-to-run variation can occur within the same condition.


3. What would happen with stronger future backbones?

The review asks whether substantially stronger future models such as GPT-6 or Opus-5 would achieve higher scores, and which dimensions would benefit most. While we do not make claims about unevaluated models, Table 4 already provides a controlled backbone comparison: CoI-Agent is fixed while only the backbone changes.

The existing results reveal three consistent patterns.

(a) Resource-constraint understanding.
Stronger backbones are better at constructing experimental plans that satisfy the explicit compute, API, time, and human-resource constraints. In contrast, weaker models such as Llama 4 Scout obtain substantially lower Budget Feasibility scores, indicating difficulty translating an idea into a resource-consistent research plan.

(b) Research-context grounding and problem formulation.
Stronger backbones also perform better in Context Grounding and Problem Formulation. This suggests that model capability affects the ability to understand the supplied scholarly context, use relevant prior knowledge, and formulate a research question that is well grounded in that context.

(c) Idea novelty.
Weaker models tend to obtain substantially lower novelty scores, while stronger backbones generally perform better across the problem, method, scenario, and insight novelty dimensions. The improvement is not necessarily uniform or monotonic across every individual dimension, but the low-capability models clearly underperform on novelty overall.

Takeaway.
Our current evidence therefore suggests that stronger backbone capability is generally associated with improvements in three aspects of research ideation: resource-constraint understanding, research-context grounding, and idea novelty. We do not assign a hypothetical score to GPT-6 or Opus-5 without evaluating them; rather, AlphaIdeaBench is designed to measure exactly which aspects improve as stronger future models become available.

We will incorporate the new human-validation and repeated-budget results in the revision and clarify the scope of the corresponding conclusions.

Dear Reviewer,

As the rebuttal period is approaching its end, we would greatly appreciate it if you could let us know whether our response and additional analyses have addressed your concerns. If so, we would also be grateful if you could reconsider your score in light of the clarification and new evidence.

Thank you again for your time and consideration.

Reviewer_91X7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AlphaIdeaBench, a benchmark of 100 research ideation tasks drawn from workshop calls and survey open problems, with explicit resource budgets. It evaluates proposals across 12 dimensions of validity, novelty, and feasibility, comparing eight harnesses and eight backbone models. The analysis examines how models, harnesses, and budgets affect the proposed research.

Critical strengths
  1. The benchmark provides shared task contexts, resource budgets, and detailed rubrics for comparing research proposals across multiple dimensions.

  2. The experiments compare both harness and backbone models, with additional analyses of problem selection and how proposals change under different budgets.

Critical weaknesses
  1. The paper does not clearly establish its contribution beyond prior work on evaluating research ideas and agent-generated research [1,2,3]. These studies already examine research quality, differences between agents, and resource constraints. The authors should explain what new insights emerge from focusing on proposals and introducing finer-grained rubrics, beyond comparing a different set of agents.

  2. The evaluation needs stronger validation. The same Gemini model scores novelty and extracts reference scores from human reviews, without reported human checks of either step. Their agreement may therefore reflect the model’s preferences rather than human judgments. The authors should have human experts independently score a subset and report their agreement with Gemini, while also checking whether the extracted reference scores faithfully represent the original reviews.

[1] Zhang et al. How Far Are We From True Auto-Research?

[2] Chen et al. MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research.

[3] Si et al. Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers.

Questions for authors

See Weaknesses.

4 comments

Thank you for the constructive review. We appreciate the reviewer’s recognition of the shared task contexts, explicit resource budgets, and fine-grained evaluation rubric. Following the review, we also conducted additional human validation experiments. These new analyses strengthen the reliability of AlphaIdeaBench, while the benchmark design itself remains our primary contribution.

1. Contribution beyond prior work

We do not claim that AlphaIdeaBench is the first work to evaluate research ideas, agents, or research feasibility. The distinction lies in what is being evaluated, how resources enter the evaluation, and what diagnostic information the benchmark provides.

(a) Pre-execution research proposals as the evaluation object.
Prior work studies research ideation or broader automated-research pipelines in different forms. AlphaIdeaBench deliberately isolates the pre-execution research proposal. We evaluate whether the research decision itself is scientifically valid, novel, and feasible before implementation, experimentation, debugging, and paper writing introduce additional sources of variation.

This separation allows the benchmark to answer a narrower but important question: given the same research context and resources, how good is the research plan proposed by the ideation system itself?

(b) Explicit quantitative resource envelopes as a controlled input.
Prior work may consider feasibility or computational resources, but AlphaIdeaBench makes a quantitative resource envelope an explicit input to every ideation task, including compute, API budget, time, and human effort.

All systems receive the same scholarly context, resource envelope, output schema, and evaluation rubric. The Academic setting represents a conventional research budget and is deliberately fixed as a control variable, so that differences in proposals are not confounded by different resource allocations.

We further vary this resource envelope in controlled experiments to study how the proposed research changes when resources become more or less restrictive. Thus, budget is not only a feasibility criterion; it is also an experimental variable in the benchmark.

(c) Fine-grained evaluation provides diagnostic rather than only aggregate comparison.
The 12 dimensions are designed to reveal where an ideation system succeeds or fails rather than only assigning a single research-quality score. In particular, novelty is decomposed into problem, method, scenario, and insight novelty, while validity and feasibility are separately evaluated.

This decomposition reveals several findings that would be obscured by an aggregate metric:

  1. With a strong shared backbone, validity and feasibility largely saturate, while differences between harnesses concentrate more strongly in novelty.
  2. Scenario novelty remains a common bottleneck across the evaluated systems, even when other novelty dimensions are substantially higher.
  3. Changing the backbone affects not only novelty, but also research grounding, problem formulation, and the ability to satisfy explicit resource constraints.

These observations show that AlphaIdeaBench is not simply comparing a different collection of agents; it provides a controlled setting for diagnosing which component of research ideation limits current systems.

We will revise the Introduction and Related Work to make these distinctions from [1–3], and the resulting empirical insights, substantially more explicit.

2. Human validation of the evaluation

The review raises two important validation questions: whether Gemini’s reviewer-to-score mapping faithfully reflects human judgments, and whether the production judge itself agrees with humans on the complete proposals evaluated by AlphaIdeaBench. We conducted independent human studies for both stages, with annotators blind to the AI scores.

(a) Human validation of reviewer-comment-to-score mapping.
We sampled 30 papers, with 10 each from ICLR, ICML, and NeurIPS. Three experts independently read the reviewer comments and assigned scores on the same four novelty dimensions.

ExpertWithin ±1Exact
199.17%82.5%
2100%87.5%
399.17%83.3%

Thus, exact agreement between the AI-derived reference score and human experts is 82.5–87.5%, while essentially all judgments are within one point. This provides a direct human check of the reviewer-to-score conversion rather than relying on Gemini self-consistency.

(b) Human evaluation on complete agent-generated proposals.
We additionally evaluated the judge on the benchmark’s actual evaluation object. We sampled 30 complete agent-generated proposals, with 15 from each task source, and asked the same three experts to independently score all 12 dimensions.

ExpertWithin ±1Exact
188.9%80.0%
291.9%80.3%
387.2%83.6%

The high agreement therefore also holds on full structured proposals, addressing the concern that validation on published-paper titles/abstracts may not transfer to the actual AlphaIdeaBench outputs.

(c) Direct faithfulness audit against the original reviews.
We further sampled another 60 papers, 20 from each venue, and asked three experts to judge whether the generated score and rationale faithfully represented the original reviewer comments. The three experts judged 93.3%, 94.6%, and 93.8% of mappings as faithful.

As complementary evidence, we embedded every sentence of the original review using gemini-embedding-001 and matched each generated rationale to its closest review sentence. The mean maximum semantic similarity is 0.874 overall (ICLR: 0.875; ICML: 0.874; NeurIPS: 0.873). We treat the human audit as the primary evidence and this semantic analysis as an additional grounding check.

(d) Cross-family evidence.
The manuscript already includes additional cross-model validation: Table 2 compares multiple model families on the 600-paper alignment set, and Appendix I independently re-scores the generated proposals with GPT-5 mini. We will reorganize these results together with the new human studies to make the validation pipeline clearer.

Takeaway.
These new experiments independently validate both stages raised in the review: (1) converting human reviewer comments into novelty reference scores, and (2) using the judge to evaluate complete agent-generated proposals. They substantially reduce the concern that the reported agreement is driven only by Gemini’s own preferences.

Thank you for your reply. I think my concerns have been addressed. Therefore, i will increase my score.

Thank you very much for your follow-up and for reconsidering the score. We greatly appreciate your constructive feedback and are glad that our clarification and additional analyses addressed your concerns.

Reviewer_zfW5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AlphaIdeaBench, a benchmark of 100 research-ideation tasks drawn from workshop calls for papers and survey open problems. Each task has a fixed resource budget and a shared proposal schema. An LLM judge scores proposals on 12 dimensions covering validity, novelty, and feasibility, and its novelty scores are validated against reviewer comments on 600 ML conference papers. The authors compare eight harnesses and eight backbones. They find that CoI-Agent ranks highest, that scenario novelty is low across all systems, and that the backbone and the harness shape different aspects of the proposals.

Critical strengths
  • The paper addresses a real gap. Ideation agents are usually evaluated with their own prompts and judges, without an explicit resource constraint. Scoring feasibility against a stated budget is a useful and underexplored idea.
  • The rubric is fine-grained and released with descriptors and checklists. Splitting novelty into problem, method, scenario, and insight is more informative than a single novelty score, and the attempt to ground the judge in real peer-review text is a step in the right direction.
  • The empirical study is broad, covering eight harnesses and eight backbones, and produces several interesting observations: scenario novelty is consistently low, and the backbone and the harness appear to influence different aspects of the proposal.
Critical weaknesses
  • The judge's reliability is not adequately established.

    • Circular validation. The novelty "ground truth" is produced by Gemini 3.7 Flash mapping reviewer comments to scores, and the judge being validated is the same model. Agreement may partly reflect self-consistency, and the mapping is never checked against human labels.
    • Mismatch with the main evaluation. Validation uses only the titles and abstracts of published papers, while the main evaluation scores full agent-generated proposals. These differ in context, writing style, and type of object. The claimed symmetry also does not hold, since reviewers read the full paper.
    • A human evaluation on agent proposals and cross-family judges are needed.
  • The experiments do not sufficiently support the claims. Each task is generated and scored only once, with no variance or significance tests, yet CoI-Agent (4.31) is declared best over SciMON (4.29). Eight of the twelve dimensions are near ceiling, which compresses Overall. The harness-vs-model conclusion relies on two one-way sweeps rather than a factorial design, so interactions are untested, and the challenge-selection analysis depends on an unvalidated keyword heuristic.

Questions for authors
  • Why do all 100 tasks share one academic budget, with other tiers tested only on three topics? Topic-specific budgets based on domain and experimental cost seem more natural, and a uniform budget may favor cheap topics on feasibility.

  • With all harnesses scoring 4.02–4.31 Overall and validity and feasibility near ceiling, how well does the benchmark discriminate between harnesses? How much of the gap exceeds run-to-run variance?

  • How correlated are the twelve dimensions? Are the validity and feasibility dimensions nearly collinear for strong backbones, and is there evidence that they are "orthogonal axes"?

4 comments

Thank you for the constructive review and for recognizing the explicit resource constraints, fine-grained rubric, and broad comparison across harnesses and backbones. Following the review, we conducted additional validation and robustness analyses to strengthen the empirical reliability of AlphaIdeaBench.

1. Judge reliability and human validation

The review raises three concerns: (i) potential self-consistency from using Gemini in both reviewer-score mapping and judging, (ii) whether mapped scores faithfully reflect human reviews, and (iii) whether validation on published papers transfers to complete agent-generated proposals. We address each below.

(a) Independent human validation of reviewer-to-score mapping.
We sampled 30 papers, 10 each from ICLR, ICML, and NeurIPS. Three experts independently read the reviewer comments and scored the same four novelty dimensions, blind to the AI-generated scores.

ExpertWithin ±1Exact
199.17%82.5%
2100%87.5%
399.17%83.3%

More than 82% of human judgments exactly match the mapped scores, and essentially all are within one point, providing an independent check beyond Gemini self-consistency.

(b) Human evaluation on the actual benchmark outputs.
Because the original alignment analysis uses published-paper titles/abstracts, we additionally sampled 30 full agent-generated proposals (15 from each task source). The same three experts independently scored all 12 dimensions, again blind to the AI scores.

ExpertWithin ±1Exact
188.9%80.0%
291.9%80.3%
387.2%83.6%

Agreement remains high on the complete proposals used by the benchmark, not only published-paper abstracts.

(c) Faithfulness to the original reviewer comments.
We further sampled 60 papers (20 per venue) and asked three experts whether each generated score and rationale faithfully represented the original review. Agreement rates were 93.3%, 94.6%, and 93.8%.

As a complementary check, we embedded each review sentence using gemini-embedding-001 and matched each rationale to its closest sentence. Mean maximum similarity is 0.874 overall (ICLR: 0.875; ICML: 0.874; NeurIPS: 0.873). We treat the expert audit as the primary evidence.

(d) Cross-family validation.
The manuscript also includes cross-family evidence: Table 2 compares multiple model families on the 600-paper alignment set, while Appendix I re-scores generated proposals using GPT-5 mini. We will reorganize these results with the new human studies to clearly separate reviewer-comment mapping from full-proposal judging.

Takeaway.
The added human studies directly test both potential failure points: reviewer-to-score mapping and judge alignment with humans on complete proposals.


2. Discrimination between CoI-Agent and SciMON

The review notes that the Overall difference between CoI-Agent (4.31) and SciMON (4.29) is small and asks how it compares with run-to-run variability.

(a) Robustness to run-to-run variation.
We fixed a random subset of 20 tasks (10 workshop and 10 survey) and independently reran both CoI-Agent and SciMON three times per task under the same Academic budget and Gemini 3.7 Flash backbone, yielding 120 new proposals.

RunCoI-AgentSciMONDifference
14.15424.1375+0.0167
24.14174.0708+0.0709
34.14174.0833+0.0584

Across the three runs, CoI-Agent averages 4.1459 (SD = 0.0072) and SciMON 4.0972 (SD = 0.0355), with a mean gap of 0.0487. CoI-Agent ranks higher in all three runs.

(b) Interpretation of the small Overall gap.
The 12-dimensional rubric is intended not only for aggregate ranking but also to reveal system-level differences. SciMON scores higher on Method Novelty (3.95 vs. 3.38) and Insight Novelty (3.78 vs. 3.40), whereas CoI-Agent scores higher on Scenario Novelty (1.87 vs. 1.47) and Budget Feasibility (4.86 vs. 4.27).

We therefore avoid treating the small Overall gap as strong evidence of universal superiority. Instead, CoI-Agent has the highest observed Overall score, while the fine-grained rubric reveals distinct strengths and weaknesses obscured by the aggregate mean.

3. Why use one shared Academic budget?

The common Academic budget is an intentional experimental control rather than an assumption that all research topics intrinsically require the same resources.

(a) The budget represents a conventional academic research setting.
The Academic setting was designed as a common reference resource envelope for comparing ideation systems under the same constraints.

(b) The budget is fixed to control the comparison.
All systems receive the same compute, API, time, and human-resource constraints. This prevents differences in proposal quality from being confounded by differences in resource allocation. If each topic were manually assigned a different budget, feasibility differences could arise either from the ideation system or from the topic-specific budget calibration, complicating cross-agent comparisons.

(c) Topic-dependent resource sensitivity is examined separately.
The purpose is not to claim that every topic naturally requires identical resources. Instead, the main benchmark holds resources fixed, while the additional three-topic experiment explicitly varies the resource envelope to examine how proposals respond to resource availability. These topics cover different resource profiles and provide a separate sensitivity analysis of how resource constraints affect proposed research.

We will clarify in the revision that the shared Academic budget serves as a common experimental control, while the multi-budget experiment separately studies sensitivity to the resource envelope.


### 4. Correlation among the 12 dimensions

At around Line 200 of the original manuscript, “orthogonal axes” was intended to describe conceptually distinct and non-compensatory aspects of proposal quality: for example, novelty cannot compensate for hallucinated grounding, nor can an ambitious formulation bypass resource constraints. It was not intended to imply zero statistical correlation.

Following the reviewer’s question, we computed Spearman correlations over all 800 proposals (100 tasks × 8 harnesses) from the main experiment. At the pillar level, Validity–Feasibility is 0.272, Validity–Novelty is 0.104, and Novelty–Feasibility is −0.012. We further report the complete dimension-level analysis below.

D1–D4 denote Context Grounding, Problem Formulation, Motivation–Method Consistency, and Testability; D5–D8 denote Problem, Method, Scenario, and Insight Novelty; D9–D12 denote Technical, Data, Budget, and Verification Feasibility.

D1D2D3D4D5D6D7D8D9D10D11D12
D11.000
D2.3821.000
D3.471.6791.000
D4.346.312.2781.000
D5.003.118.050.0521.000
D6.126.039.064.189.2201.000
D7−.070.044.021−.042.334−.1401.000
D8.066.147.088.056.372.464.0011.000
D9.334.282.268.362−.069.006−.036−.0011.000
D10.108.095.169.038−.062.092−.176−.001.0621.000
D11.045.076.096−.081.062−.080.049−.002.091.0371.000
D12.382.494.455.459.028.091.011.093.471.089.0631.000

The correlations show some expected relationships, but no pervasive high collinearity. The largest pairwise correlation is ρ=0.679 (D2–D3), while most cross-pillar correlations are small. In particular, Validity and Feasibility are not nearly collinear at the pillar level (ρ=0.272), and the novelty dimensions are largely distinct from both.

These results support the diagnostic purpose of the rubric: the dimensions capture related but substantially different aspects of proposal quality. To avoid ambiguity, we will replace “orthogonal axes” with “distinct but potentially correlated evaluation dimensions,” while preserving the original non-compensatory interpretation.


5. Harness/backbone and challenge-selection analysis

Our harness and backbone experiments are two controlled one-factor sweeps rather than a full harness × backbone factorial design. We therefore limit the conclusion to the observed settings: challenge selection varies more across backbones, whereas multi-challenge composition varies more across harnesses, without making claims about general interaction effects.

For challenge selection, manual inspection indicates that the current automatic assignments are sufficiently consistent with the intended challenge mappings for this descriptive analysis. We will clarify the matching procedure in the revision.

Thank you for your clarification and additional analysis, which resolve my concerns well. I'll keep my positive score.

Thank you very much for your positive feedback and for taking the time to review our clarification and additional analyses. We greatly appreciate your constructive comments, which helped us improve the paper.

Comments on the paper

3 comments
Area_Chair_pMhVOpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,

Your AC

AuthorsOpenReview ↗

Summary of Rebuttal

Dear AC and Reviewers,

Thank you for your time and constructive comments. Below we summarize the main comments, our responses, and the new results added during the rebuttal. The human validation studies (Appendix J), the repeated budget experiments (Table 5), and the correlation analysis (Appendix K) have been added to the revised PDF.

1. Strengths noted by reviewers

  • The explicit resource budget makes the evaluation closer to real research, and scoring feasibility against this budget is useful and less explored (zfW5, 91X7, kLYY).
  • The 12-dimensional rubric, especially splitting novelty into problem, method, scenario, and insight novelty, gives more information than a single score. The released descriptors and checklists make it easy to reuse (zfW5, kLYY, 91X7).
  • The experiments cover eight harnesses and eight backbones, and show findings such as low Scenario Novelty across systems and the different roles of backbone and harness (zfW5, 91X7).

2. Discussion status

  • Reviewer zfW5: concerns resolved; kept the score of 3 (Weak acceptance).
  • Reviewer 91X7: concerns addressed; raised the score to 3 (Weak acceptance).
  • Reviewer kLYY: asked further questions about the human study (inter-annotator agreement, annotator expertise, and sample representativeness). We have answered all of them in our follow-up response (see C1).

3. Main concerns and responses

C1. Reliability of the judge (zfW5, 91X7, kLYY). Since Gemini 3.7 Flash is used both to map reviewer comments to novelty scores and as the judge, we conducted three human studies, with annotators blind to the AI scores (Appendix J):

StudyScaleExact / FaithfulWithin ±1
Reviewer comments → 4 novelty scores30 papers, 3 experts82.5–87.5%99.17–100%
Full agent proposals → 12-D scores30 proposals, 3 experts80.0–83.6%87.2–91.9%
Faithfulness of score and rationale to the original reviews60 papers, 3 experts93.3–94.6%—

In the follow-up response to kLYY, we also reported the agreement among the three experts on the 30 papers: 81.1% exact and 98.3% within ±1, averaged over expert pairs. All 120 paper–dimension items were annotated by all three experts. The agreement between the experts and the AI-derived scores is at least as high as the agreement among the experts. The 30 papers cover all nine research areas of the benchmark, with a distribution close to that of the benchmark. Table 2 and Appendix I (re-scoring with GPT-5 mini) provide additional cross-family results. The study has some limitations: it includes 30 papers, only papers whose reviews discuss all four novelty facets are included, and the annotators were not matched to subfields.

C2. CoI-Agent vs. SciMON under run-to-run variance (zfW5). We reran both systems three times on 20 fixed tasks (120 new proposals). CoI-Agent scored higher in all three runs (mean 4.146 vs. 4.097, gap +0.049), but the gap is small. We therefore only claim that CoI-Agent has the highest observed Overall score, not that it is better in general. The two systems also differ clearly at the dimension level:

  • SciMON is higher on Method Novelty (3.95 vs. 3.38) and Insight Novelty (3.78 vs. 3.40).
  • CoI-Agent is higher on Scenario Novelty (1.87 vs. 1.47) and Budget Feasibility (4.86 vs. 4.27).

C3. Budget effect vs. generation variance (kLYY). We generated three more proposals for each of the 3 topics × 3 budgets (27 new proposals; Table 5 in Section 4).

  • Mean Scenario Novelty increases with budget for VLA (1.33 → 2.33 → 3.00) and interpretability (1.00 → 3.33 → 4.00).
  • It does not increase for tool use (2.00 → 1.33 → 1.00).
  • The sample variance within each condition is 0 or 0.33.

We have revised the claim: the effect of the budget depends on the topic, and it appears when resources actually limit the experimental setting. A single high-scoring proposal, such as the tool-use case with Scenario Novelty 4 in Appendix E.3, should not be taken as a budget effect.

C4. Correlation among the 12 dimensions (zfW5). We computed Spearman correlations over all 800 proposals in the main experiment (Appendix K, Table 21). At the pillar level, ρ is 0.272 for Validity–Feasibility, 0.104 for Validity–Novelty, and −0.012 for Novelty–Feasibility. The largest pairwise correlation is 0.679, between Problem Formulation and Motivation–Method Consistency. We have changed "orthogonal axes" to "distinct but potentially correlated dimensions" in Section 3.2.

C5. Scope of the harness/backbone conclusions and the shared budget (zfW5). The harness and backbone experiments are two one-factor sweeps, not a factorial design. Our conclusions therefore hold only for the evaluated settings, and we do not claim anything about interaction effects. All tasks use the same Academic budget so that differences between systems are not caused by different budgets. The effect of the budget is studied separately in the multi-budget experiment.

C6. Contribution compared with prior work (91X7). We do not claim to be the first work on evaluating research ideas. Our contribution is the combination of three parts:

  • evaluating the proposal before execution;
  • an explicit quantitative resource budget given as input to every task;
  • a 12-dimensional rubric for diagnosis.

With this setup we can observe things that a single overall score does not show. For example, with strong backbones, validity and feasibility are close to the ceiling; Scenario Novelty is low for all systems; and systems with similar Overall scores can have very different profiles.

C7. Stronger backbones (kLYY). In our backbone experiment (Table 4), stronger backbones score higher on Budget Feasibility, Context Grounding, Problem Formulation, and novelty. We do not predict scores for models we have not evaluated.

During the rebuttal, we added three human studies, 147 newly generated proposals, and a correlation analysis over the 800 main-experiment proposals, and we narrowed several claims. We hope these results address your main concerns. Thank you again for your time and engagement.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.