- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AutoSciBench automatically generates and refines benchmarks for scientific agents. It separates task design into concepts and construction recipes, then produces questions, data files, and reference answers. Solver trajectories guide task revisions, while experience from previous runs informs new task generation. Experiments use two generator models across computational biology, materials science, and clinical imaging. The authors report lower solver accuracy on generated biology and materials tasks, higher average LLM-rated quality scores, and ablations examining planning, refinement, and experience reuse.
Critical strengths
-
The framework constructs complete tasks with data and reference answers and revises them using observed solver behavior. This addresses a practical obstacle to keeping scientific-agent benchmarks up to date.
-
The experimental results show that generated tasks also challenge models that did not participate in generation. This supports their usefulness beyond the specific model used for refinement.
Critical weaknesses
-
Materials-science verifiability drops from 4.76 to 3.49/3.93. Although generation includes verification, the separate quality judges cannot inspect reference answers or execute solutions. Thereofore, the claim that increased difficulty preserves task reliability is therefore not well supported.
-
Tasks are labeled hard after zero successes in three trials, and these outcomes guide refinement and selection. Independent retesting of initial and selected tasks is not reported. The gains may therefore partly reflect selection of unlucky solver runs.
Questions for authors
- What kinds of independent checks were performed on the final reference answers?
- Were refinement results evaluated with fresh trials? How were tasks that stopped early included in the curves?
3 comments
Initial Response by Authors (2/2)
Comment 2: Independent retesting of initial and selected tasks is not reported. How were tasks that stopped early included in the curves? The gains may therefore partly reflect selection of unlucky solver runs.
Response 2: Thank you for your constructive comment. In the original refinement curves, the last difficulty ratings of tasks that stopped early after being judged hard and verifiable were carried forward to subsequent rounds.
We further re-evaluate 10 materials-science tasks generated by Claude across refinement rounds. For tasks that stopped early, this re-evaluation uses the unchanged QA with three independent solver trials at each subsequent round. As shown in Table R4, the solving ratio decreases, while the proportion of hard tasks increases, suggesting that the difficulty gains from iterative refinement are also observed under independent re-testing. In response to your comment, we have added these independent re-evaluation results as Fig. 10 in the revised paper.
| Refinement Round | Solving Ratio (%) | Hard Tasks (%) |
|---|---|---|
| 0 | 100.0 | 0.0 |
| 1 | 100.0 | 0.0 |
| 2 | 83.3 | 10.0 |
| 3 | 80.0 | 10.0 |
| 4 | 70.0 | 20.0 |
| 5 | 80.0 | 20.0 |
| 6 | 60.0 | 30.0 |
| 7 | 56.7 | 30.0 |
| 8 | 63.3 | 30.0 |
| 9 | 63.3 | 30.0 |
Table R4: Independent re-evaluation of Claude-generated materials-science tasks across refinement rounds.
Comment 3: What kinds of independent checks were performed on the final reference answers?
Response 3: The refinement procedure leverages verifiability judgement to check whether the question and provided data support a unique, well-defined answer. Specifically, the judge reviews the reference answer together with the question, data, and solver trajectories, assessing whether the target is sufficiently specified and whether disagreements arise from task ambiguity or incorrect solver analyses. When the judge identifies ambiguity or inconsistencies, its feedback guides revisions to the recipe or concept, followed by task re-generation via concept or recipe revision. For the final benchmark, we select only tasks judged verifiable.
We thank the reviewer for your time and feedback and hope that our responses have addressed your concerns. If you have any further questions, we are happy to address them.
Brief Summarization of Rebuttal for Reminder
Dear Reviewer xDWE,
We sincerely appreciate your time and constructive feedback. To facilitate your review, we briefly summarize our rebuttal below.
C1-1. Materials-science verifiability
We analyze representative cases to understand the verifiability gap. The high verifiability of human-curated materials-science tasks mainly reflects directly checkable answers, even when these tasks require little scientific analysis.
C1-2. Reliability of quality assessment
To strengthen verification, we assessed materials-science and clinical-imaging tasks with reference-answer access and code execution. Generated benchmarks received higher average quality scores than human-curated benchmarks under these more rigorous assessment settings.
C2. Effectiveness of iterative refinement: Independent retesting of initial and selected tasks
To address the possibility of unlucky solver runs, we re-evaluated materials-science tasks across refinement rounds using fresh, independent trials. These results support the effectiveness of our refinement procedure in increasing task difficulty even under independent evaluation.
We sincerely thank you again for your constructive feedback and hope that the additional results and clarifications have helped address your concerns. As the discussion period approaches its end, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your main concerns.
Best regards,
Authors
Initial Response by Authors (1/2)
We sincerely thank you for your constructive comments and positive feedback on:
We initially address your concerns below.
Comment 1-1: Materials-science verifiability drops from 4.76 to 3.49/3.93. Therefore, the claim that increased difficulty preserves task reliability is therefore not well supported.
Response 1-1: We would like to clarify that the high verifiability scores of human-curated materials-science tasks are due to their directly checkable answers. The evaluated MatTools tasks use a multiple-choice code-completion format, allowing the answers to be verified against the provided source code. For example, both judges assign the task in Figure 8 of the revised paper a verifiability score of 5, while assigning usefulness and fitness scores of 1 because it provides a directly checkable answer and requires little scientific analysis. In contrast, human-curated tasks in other domains and generated tasks require agents to compute scientific quantities from data and interpret the results, which is reflected in their higher usefulness and fitness scores. We have revised Section 4.3 to distinguish these improvements from the lower verifiability scores in materials science.
Comment 1-2: Although generation includes verification, the separate quality judges cannot inspect reference answers or execute solutions.
Response 1-2: We appreciate your valuable comment. Due to the absence of the reference answers of some benchmarks such as CompBioBench, we did not provide the reference answers as inputs, while focusing on the quality assessment by read-only inspection tools. For the material science and clinical imaging, we further conduct quality assessment with reference answers and the code execution to more rigorously evaluate the task quality. As shown in Table R1, 2, and 3, Claude- and GPT-generated tasks show higher average quality scores compared to the human-curated tasks, indicating that the overall comparison remains consistent regardless of whether quality judges have access to reference answers or the ability to execute codes.
Table R1: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol without reference answers or code execution.
Table R2: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol without reference answers, with code execution enabled.
Table R3: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol with reference answers and code execution enabled.
Due to the character limit, the response is continued below.