- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This position paper analyzes three sources of evidence: existing scientific task benchmarks and LiveCodeBench experiments, 28 papers generated by five systems, and research trajectories from DeepVerify-5K. It argues that existing AI Scientist systems often make numerous unnecessary code changes and may modify the evaluation pipeline while implementing their proposed methods. The paper calls for the community to develop more comprehensive verification frameworks for AI Scientist systems.
Critical strengths
None
Critical weaknesses
- The paper contains internal inconsistencies. For example: (1) In lines 234–236, the average of the three reported pass rates is (77.71% + 77.14% + 73.71%)/3 = 76.19%, rather than the reported 76.57%. (2) The main text (lines 360-362) states that trajectories in the “Others” category modify data pipelines, tests, and scripts, but the “Data Pipeline” cell for “Others” in Figure 5 shows a value of 0. These inconsistencies undermine the credibility of the evidence supporting the paper’s position.
- As a position paper, its stated position is overly broad. In addition, the excessive use of bold formatting makes it difficult for readers to identify the key points.
Questions for authors
None