- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AlphaIdeaBench, a benchmark for evaluating the automated research idea generation capability. It does not focus on whether the agent can produce "paper-like" texts, but rather: under specific constraints of computing power, budget, time, and human input, can it propose scientifically valid, novel, and feasible research plans.
Critical strengths
-
Explicit resource constraints make the evaluation practically relevant. Research ideas must be feasible within available compute, time, funding, and data constraints, rather than merely novel on paper. AlphaIdeaBench explicitly incorporates these constraints and assesses whether the full experimental plan—including baselines, ablations, and repeated runs—fits the stated resource budget. This makes the evaluation more practically relevant than approaches that assess ideation without accounting for resource limitations.
-
The fine-grained rubric distinguishes different forms of scientific contribution. The twelve-dimensional rubric separates validity, novelty, and feasibility. In particular, decomposing novelty into problem, method, scenario, and insight novelty helps distinguish methodological improvements within an unchanged research setting from contributions that introduce new questions, scenarios, or mechanistic insights. The publicly available scoring descriptors and checklists also make the evaluation protocol easier to reuse.
Critical weaknesses
- The use of real reviewer comments provides a meaningful human reference. However, Gemini both converts these comments into axis-specific reference scores and serves as the judge being evaluated. The reliability of this conversion is insufficiently established, leaving uncertainty about how much the reported agreement reflects human judgments rather than shared model biases.
- The budget study does not adequately distinguish budget effects from generation variability. It covers only three topics, with one proposal per topic–budget condition and no reported within-condition repeats. Appendix E.3 illustrates the concern: rerunning the tool-use topic under the same 200 GPU-hour budget produces a different proposal, with scenario novelty dropping from 4 to 2. Thus, the examples demonstrate possible changes across budgets but do not establish that these are systematic responses to resource constraints rather than run-to-run variation. This limits confidence in a central empirical finding; repeated generations and uncertainty estimates are needed to substantiate it.
Questions for authors
1.How was the mapping from reviewer comments to the four novelty scores validated to ensure that it faithfully reflects reviewer judgments rather than the mapping model’s own interpretations? 2. Were any repeated generations conducted for the same topic–budget condition beyond the rerun described in Appendix E.3? If so, how does within-condition variability compare with the differences across budgets? This would help determine whether the reported changes reflect systematic budget responses rather than run-to-run variation. 3. If the model being evaluated is replaced with a more powerful one, such as GPT-6 or Opus-5, will their scores on the list be significantly higher than those of the current weaker models being tested? Which aspect will be stronger?
8 comments
(c) Topic-dependent budget responses.
The repeated runs show different patterns across topics. For VLA and interpretability, the mean Scenario Novelty scores increase monotonically across the three resource levels:
- VLA: 1.33 → 2.33 → 3.00.
- Interpretability: 1.00 → 3.33 → 4.00.
The corresponding low-to-high differences are 1.67 and 3.00 points. In contrast, tool use shows no positive trend (2.00 → 1.33 → 1.00). Moreover, the original score-4 AsyncTx-Agent was not reproduced in the subsequent generations, indicating that an individual high-scoring proposal should not be interpreted as a systematic budget effect.
Takeaway.
The repeated-generation analysis therefore refines our original conclusion. We do not claim that increasing the resource budget always increases Scenario Novelty. Instead, the observed response to the resource envelope is topic-dependent: the repeated-run mean increases monotonically for VLA and interpretability, but not for tool use, while non-negligible run-to-run variation can occur within the same condition.
3. What would happen with stronger future backbones?
The review asks whether substantially stronger future models such as GPT-6 or Opus-5 would achieve higher scores, and which dimensions would benefit most. While we do not make claims about unevaluated models, Table 4 already provides a controlled backbone comparison: CoI-Agent is fixed while only the backbone changes.
The existing results reveal three consistent patterns.
(a) Resource-constraint understanding.
Stronger backbones are better at constructing experimental plans that satisfy the explicit compute, API, time, and human-resource constraints. In contrast, weaker models such as Llama 4 Scout obtain substantially lower Budget Feasibility scores, indicating difficulty translating an idea into a resource-consistent research plan.
(b) Research-context grounding and problem formulation.
Stronger backbones also perform better in Context Grounding and Problem Formulation. This suggests that model capability affects the ability to understand the supplied scholarly context, use relevant prior knowledge, and formulate a research question that is well grounded in that context.
(c) Idea novelty.
Weaker models tend to obtain substantially lower novelty scores, while stronger backbones generally perform better across the problem, method, scenario, and insight novelty dimensions. The improvement is not necessarily uniform or monotonic across every individual dimension, but the low-capability models clearly underperform on novelty overall.
Takeaway.
Our current evidence therefore suggests that stronger backbone capability is generally associated with improvements in three aspects of research ideation: resource-constraint understanding, research-context grounding, and idea novelty. We do not assign a hypothetical score to GPT-6 or Opus-5 without evaluating them; rather, AlphaIdeaBench is designed to measure exactly which aspects improve as stronger future models become available.
We will incorporate the new human-validation and repeated-budget results in the revision and clarify the scope of the corresponding conclusions.
Dear Reviewer,
As the rebuttal period is approaching its end, we would greatly appreciate it if you could let us know whether our response and additional analyses have addressed your concerns. If so, we would also be grateful if you could reconsider your score in light of the clarification and new evidence.
Thank you again for your time and consideration.
Thank you for the constructive review and for recognizing the practical value of explicit resource constraints and the diagnostic role of the fine-grained rubric. Following the review, we conducted additional human-validation and repeated-generation experiments to directly examine the two main empirical concerns. We also clarify the implications of our existing backbone comparison for stronger future models.
1. Faithfulness of reviewer-comment mapping
The review asks whether the four novelty reference scores extracted from human reviewer comments faithfully reflect the reviewers’ judgments rather than Gemini’s own interpretation. We added two independent human checks.
(a) Independent human scoring of reviewer comments.
We sampled 30 papers, with 10 each from ICLR, ICML, and NeurIPS. Three experts independently read the original reviewer comments and assigned scores on the same four novelty dimensions. The annotators were blind to all AI-generated scores.
The exact agreement between the AI-derived reference scores and the independent human scores is 82.5–87.5%, while essentially all judgments are within one point. This provides a direct human validation of the reviewer-comment-to-score mapping rather than relying only on Gemini self-consistency.
(b) Direct audit of score/rationale faithfulness.
We further sampled another 60 papers, with 20 from each venue, and asked three experts to directly judge whether the generated score and its accompanying rationale faithfully represented the original reviewer comments. The corresponding faithfulness rates were 93.3%, 94.6%, and 93.8%.
As complementary evidence, we embedded each sentence of the original reviews using
gemini-embedding-001and matched each generated rationale to its most semantically similar source sentence. The mean maximum similarity is:We treat the expert audit as the primary evidence, while the semantic-similarity analysis provides an additional check that the generated rationales are grounded in the original review text.
Takeaway.
These two studies independently validate both the assigned novelty scores and the rationales used to derive them, substantially reducing the concern that the reviewer-derived reference scores primarily reflect the mapping model’s own preferences.
2. Budget effects versus generation variability
The review correctly identifies that the original budget study used one proposal per topic–budget condition, making it difficult to separate systematic budget effects from generation variability. Following this suggestion, we repeated every condition.
(a) Clarifying the 4→2 tool-use case.
We first clarify the two tool-use proposals generated under the same Academic (200 GPU-hour) condition. AsyncTx-Agent received a Scenario Novelty score of 4 because it introduced an asynchronous operating setting involving “cross-task contamination with out-of-order network latency perturbations, sibling-task hijacking, and speculative state corruption.” The judge further noted that these constraints “necessitate targeted architectural redesign,” including transactional isolation, two-phase staging, and branch-level speculative rollback.
In contrast, the same-budget rerun, TaintTool-Graph, received 2 because its TaintTool-Bench setting “builds directly on top of existing InjecAgent and BIPIA attack taxonomies,” and the judge characterized it as “the standard academic benchmark setup for evaluating indirect prompt injection defenses and schema-compliant tool calling.”
Thus, the score difference reflects substantively different scenarios generated under the same resource condition, confirming the reviewer’s concern that run-to-run generation variability can matter.
(b) Repeated-generation results.
To examine this variability more systematically, we conducted three additional full CoI-Agent generations for each of the 3 topics × 3 budget conditions, yielding 27 additional proposals. The table below reports statistics for these additional runs only; the original proposals and the previously reported tool-use rerun are reported separately so that the new repeated-generation experiment is clearly identified.
Values are mean (sample variance) over three additional generations.
Among the additional runs, the within-condition sample variance is either 0 or 0.33.