acceptodds
Under review as a conference paper at ICLR 2027

AI Scientists Need Stronger Verification Framework to Address Systemic Fragility

Abstract

Over the past few years, the development of AI Scientists has accelerated rapidly, demonstrating growing potential for automatic research across multiple disciplines. Despite this progress, the capability boundaries of current AI Scientist systems remain insufficiently characterized, especially with respect to verification ability. In this paper, we argue that autonomous verification should be treated as a first-class challenge in the development and evaluation of AI Scientist systems. Current AI Scientist systems exhibit clear verification shortcomings, demanding more comprehensive strategies to address the systemic fragility of autonomous verification. We support our position with three levels of evidence. First, we review coarse-grained evidence from existing benchmarks on complex scientific tasks, which shows that frontier models remain unreliable when tasks require domain-specific reasoning. Second, we conduct an output-level evaluation of 28 research papers generated by five AI Scientist systems, revealing persistent weaknesses in experimental design and methodological clarity. Third, we introduce DeepVerify-5K, which contains 4,890 end-to-end scientific discovery trajectories. Each trajectory provides records of idea generation, implementation, execution, and iterative revision, enabling process-level analysis of how autonomous verification fails. Our analysis of DeepVerify-5K shows that current AI Scientist systems often exhibit fragile verification behaviors, including redundant code changes and entanglement with evaluation pipelines. We hope this position paper encourages the community to pay greater attention to the systemic fragility of autonomous verification and evaluate AI Scientist systems' capabilities in closer alignment with actual scientific scenarios.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
AI Scientist, Auto Research

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rNHXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This position paper analyzes three sources of evidence: existing scientific task benchmarks and LiveCodeBench experiments, 28 papers generated by five systems, and research trajectories from DeepVerify-5K. It argues that existing AI Scientist systems often make numerous unnecessary code changes and may modify the evaluation pipeline while implementing their proposed methods. The paper calls for the community to develop more comprehensive verification frameworks for AI Scientist systems.

Critical strengths

None

Critical weaknesses
  1. The paper contains internal inconsistencies. For example: (1) In lines 234–236, the average of the three reported pass rates is (77.71% + 77.14% + 73.71%)/3 = 76.19%, rather than the reported 76.57%. (2) The main text (lines 360-362) states that trajectories in the “Others” category modify data pipelines, tests, and scripts, but the “Data Pipeline” cell for “Others” in Figure 5 shows a value of 0. These inconsistencies undermine the credibility of the evidence supporting the paper’s position.
  2. As a position paper, its stated position is overly broad. In addition, the excessive use of bold formatting makes it difficult for readers to identify the key points.
Questions for authors

None

Reviewer_Dij5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This is a position paper arguing that autonomous verification should be a first-class challenge in AI Scientist systems. The authors review existing benchmarks and conduct analyses on autonomous research trajectories to support their claim.

Critical strengths
  • The paper gives a useful position: autonomous verification is a crucial problem for building and evaluating AI Scientist systems.

  • The authors provide a good list of alternative views that documents counterarguments, allowing readers to think critically.

  • The paper is clearly written and easy to read.

Critical weaknesses
  • The authors claim a dataset contribution, but the dataset of agentic trajectories has the same setting (tasks, models, scale, and even compute) as the DeepScientist paper published at ICLR 2026. That paper also analyzes its discovery logs. What is the difference between what is released here and what is released in the DeepScientist paper?

    • Also, everything in this dataset comes from a single system, so whether the same conclusions hold on other similar data remains an open question.
  • The experiments are underdocumented. The paper gives no description of how raw logs are mapped to error categories or how the error categorization is done. The failure categories in Table 2 are also not defined anywhere.

  • This is a position paper, but the position is close to consensus and hard to argue against, which limits its value as a position.

Questions for authors
  • What is the difference between what is released here and what is released in the DeepScientist paper?

  • Why does EMPTY_OR_EXTRACT account for 69.6% of failed run instances on LiveCodeBench? Could it be due to simple regex extraction failures? Can you please provide an example?

Reviewer_9q4tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that “reliable verification” is a key bottleneck for AI Scientists. Successful code execution or improved evaluation metrics do not necessarily mean that scientific claims have been faithfully verified. The paper combines evidence from existing scientific benchmarks, LLM-based evaluation of 28 AI-generated papers, and DeepVerify-5K, a dataset of 4,890 end-to-end scientific discovery trajectories, to investigate verification failures and agents’ debugging behaviors. The analysis identifies recurring experimental and methodological weaknesses, as well as cases of agents modifying tests, run scripts, and evaluation pipelines without adequately improving the core method. The authors advocate for auditable verification processes, modular multi-agent architectures, and more efficient training feedback. The main contribution is to frame autonomous scientific verification as a distinct research challenge and provide empirical evidence and a trajectory dataset to study it. However, the validation has limitations, including potential selection bias in the reviewed papers, reliance on LLM-based reviewing rather than human peer review, and trajectories drawn from a single AI Scientist system across three computational tasks.

Critical strengths
  1. The paper identifies reliable verification as a key bottleneck for AI Scientists and supports this claim with evidence from existing benchmarks, evaluations of 28 AI-generated papers, and an analysis of scientific discovery trajectories. This combination provides a useful perspective on verification failures beyond simply measuring whether code runs or performance improves.
  2. DeepVerify-5K provides valuable process-level evidence about how AI Scientists implement ideas, debug errors, and modify tests and run scripts. This helps investigate whether agents genuinely verify scientific claims or merely make progress toward passing evaluations, offering a useful resource for future research on trustworthy autonomous science.
Critical weaknesses
  1. The empirical evidence could be strengthened to better support the paper’s broader claims about autonomous scientific verification. The evaluation of AI-generated papers relies on an LLM judge and a relatively small set of 28 publicly available papers, which may introduce selection bias. In addition, DeepVerify-5K focuses on one AI Scientist system and three computational tasks, so further evaluation across more systems and domains would help establish how broadly the findings apply.

  2. The interpretation of agents’ modifications to tests, run scripts, and evaluation pipelines could be more carefully validated. Although these behaviors may indicate potential weaknesses in verification, some modifications may also be legitimate engineering or debugging steps. Further analysis distinguishing these cases would strengthen the evidence connecting the observed behaviors to unfaithful verification.

Questions for authors
  1. How do you distinguish legitimate debugging from unfaithful verification when agents modify tests, run scripts, or evaluation pipelines? Do you have evidence that these modifications change the verification criteria or lead to misleading conclusions?
  2. To what extent do the findings from DeepVerify-5K generalize beyond the DeepScientist system and the three computational tasks studied? Could the observed verification issues be specific to this setup, or do you expect them to persist across different AI Scientist systems and research domains?
  3. How sensitive are the conclusions to the use of LLM-based reviewing? Have you checked whether the identified weaknesses in the 28 AI-generated papers are consistent with human expert evaluations?
  4. How do you validate that the observed debugging behaviors meaningfully affect scientific validity? For example, do trajectories with frequent modifications to tests or scripts show worse final scientific outcomes than those that primarily modify the core method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.