- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces SoundnessBench, a benchmark designed to evaluate whether LLMs can assess the methodological viability of research proposals before experimental execution. The authors construct a dataset of 1,099 research proposals extracted from existing ICLR submissions, using reviewer-derived soundness scores as proxy labels and additional expert annotations on 327 proposals. The benchmark evaluates 12 frontier LLMs under different prompting strategies and reports a significant optimism bias, where models frequently classify low-soundness proposals as high-soundness. The authors conclude that current LLMs are not sufficiently reliable for pre-execution scientific judgment.
Critical strengths
The paper addresses an important and timely research question regarding the ability of LLMs to evaluate scientific ideas before committing substantial computational and human resources. Such a capability would be particularly valuable for autonomous research agents, where early identification of flawed research designs could significantly improve research efficiency. The authors also make an effort to construct a relatively large benchmark covering multiple research areas and evaluate a broad range of frontier LLMs. The inclusion of different prompting strategies, additional expert annotations, and robustness analyses provides useful observations about the sensitivity of LLM judgments. Overall, I appreciate the motivation and the effort devoted to collecting and evaluating this dataset.
Critical weaknesses
-
Fundamental retrospective bias in the benchmark construction. My primary concern is that the proposed benchmark is fundamentally biased by reconstructing research proposals from completed papers. The abstract, introduction, related work, hypothesis, and experimental design of a completed manuscript have already been organized and refined based on the conducted research and its outcomes. Therefore, extracting these components and removing explicit results does not recover the original research proposal before execution. This introduces substantial hindsight bias, making the reconstructed proposals fundamentally different from the actual pre-execution research ideas that the benchmark claims to evaluate.
-
Mismatch between the benchmark labels and the research question. The paper aims to determine whether LLMs can identify methodologically flawed ideas before experimentation, yet its primary labels originate from reviewer assessments of completed research papers. These judgments may depend on experimental results, implementation quality, and evidence unavailable at the proposal stage. Although the authors introduce proposal-only expert annotations, these are still based on retrospectively reconstructed proposals and do not eliminate the underlying concern. Consequently, the benchmark does not convincingly establish that its labels represent the methodological viability of the original research ideas before execution.
-
Insufficient evidence to support the claimed conclusions. The reported optimism bias is measured against labels whose relationship to genuine pre-execution methodological soundness remains uncertain. Therefore, disagreements between LLM predictions and benchmark labels cannot necessarily be interpreted as failures of scientific judgment. The additional prompting experiments and robustness analyses are conducted on the same retrospectively constructed benchmark and do not resolve this fundamental issue. As a result, I believe the paper primarily contributes a dataset and descriptive evaluation results, without sufficient methodological evidence to support its broader conclusions about LLM research-evaluation capabilities.
Questions for authors
-
How can the authors justify reconstructing pre-execution research proposals from completed papers, given that the hypotheses, experimental designs, and related-work discussions may already have been refined based on experimental outcomes? How is this retrospective bias addressed beyond removing explicit results?
-
How can reviewer-derived soundness scores from completed papers serve as reliable ground truth for assessing methodological viability before execution? How do the authors distinguish flaws identifiable at the proposal stage from weaknesses revealed only through experimentation?
-
Have the authors considered evaluating LLMs on genuine research proposals collected before experiments are conducted, with independent expert assessments at that stage? Such an evaluation would appear more directly aligned with the paper's central research question.