acceptodds
Under review as a conference paper at ICLR 2027

AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

Abstract

As agents rapidly evolve, existing benchmarks can become saturated, limiting their ability to distinguish capabilities and reveal remaining failure modes. Particularly in scientific domains, constructing and updating benchmarks requires substantial time, labor, and domain expertise, making it difficult to keep evaluation aligned with advances in agent capabilities. We address this challenge by investigating whether scientific-agent benchmarks can be automatically generated and iteratively adapted as agent capabilities evolve. We introduce AutoSciBench, a framework that represents each task as a high-level concept specifying the scientific domain, data modality, and required reasoning approach, together with a low-level recipe specifying how the question, environment, and ground-truth answer are constructed and verified. Agents attempt to solve each task, producing solver trajectories and corresponding judge feedback which AutoSciBench uses to revise the recipe or concept, closing observed shortcuts and shifting tasks toward raw-data re-examination, interpretation of intermediate results, and evidence integration. Experience distilled from completed refinement trajectories further guides new concept generation, allowing lessons from earlier task refinement to inform subsequent benchmark construction. Starting from existing benchmarks, we evaluate AutoSciBench across computational biology, materials science, and clinical imaging. Generated benchmarks reduce average solver accuracy by 22.4 and 25.5 percentage points relative to the human-curated benchmarks in computational biology and materials science, respectively, while generated tasks receive higher average quality ratings across all three domains. Further analyses show that iterative refinement increases task difficulty and prior refinement experience guides more challenging task generation. Together, these results suggest that scientific-agent evaluation can adapt as agent capabilities advance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific agents, Automatic benchmark generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xDWEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AutoSciBench automatically generates and refines benchmarks for scientific agents. It separates task design into concepts and construction recipes, then produces questions, data files, and reference answers. Solver trajectories guide task revisions, while experience from previous runs informs new task generation. Experiments use two generator models across computational biology, materials science, and clinical imaging. The authors report lower solver accuracy on generated biology and materials tasks, higher average LLM-rated quality scores, and ablations examining planning, refinement, and experience reuse.

Critical strengths
  1. The framework constructs complete tasks with data and reference answers and revises them using observed solver behavior. This addresses a practical obstacle to keeping scientific-agent benchmarks up to date.

  2. The experimental results show that generated tasks also challenge models that did not participate in generation. This supports their usefulness beyond the specific model used for refinement.

Critical weaknesses
  1. Materials-science verifiability drops from 4.76 to 3.49/3.93. Although generation includes verification, the separate quality judges cannot inspect reference answers or execute solutions. Thereofore, the claim that increased difficulty preserves task reliability is therefore not well supported.

  2. Tasks are labeled hard after zero successes in three trials, and these outcomes guide refinement and selection. Independent retesting of initial and selected tasks is not reported. The gains may therefore partly reflect selection of unlucky solver runs.

Questions for authors
  1. What kinds of independent checks were performed on the final reference answers?
  2. Were refinement results evaluated with fresh trials? How were tasks that stopped early included in the curves?
3 comments
AuthorsOpenReview ↗

Initial Response by Authors (1/2)

We sincerely thank you for your constructive comments and positive feedback on:

  • Practical framework for constructing and updating scientific-agent benchmarks
  • Challenging tasks for models not involved in generation

We initially address your concerns below.



Comment 1-1: Materials-science verifiability drops from 4.76 to 3.49/3.93. Therefore, the claim that increased difficulty preserves task reliability is therefore not well supported.

Response 1-1: We would like to clarify that the high verifiability scores of human-curated materials-science tasks are due to their directly checkable answers. The evaluated MatTools tasks use a multiple-choice code-completion format, allowing the answers to be verified against the provided source code. For example, both judges assign the task in Figure 8 of the revised paper a verifiability score of 5, while assigning usefulness and fitness scores of 1 because it provides a directly checkable answer and requires little scientific analysis. In contrast, human-curated tasks in other domains and generated tasks require agents to compute scientific quantities from data and interpret the results, which is reflected in their higher usefulness and fitness scores. We have revised Section 4.3 to distinguish these improvements from the lower verifiability scores in materials science.


Comment 1-2: Although generation includes verification, the separate quality judges cannot inspect reference answers or execute solutions.

Response 1-2: We appreciate your valuable comment. Due to the absence of the reference answers of some benchmarks such as CompBioBench, we did not provide the reference answers as inputs, while focusing on the quality assessment by read-only inspection tools. For the material science and clinical imaging, we further conduct quality assessment with reference answers and the code execution to more rigorously evaluate the task quality. As shown in Table R1, 2, and 3, Claude- and GPT-generated tasks show higher average quality scores compared to the human-curated tasks, indicating that the overall comparison remains consistent regardless of whether quality judges have access to reference answers or the ability to execute codes.

BenchmarkVerif.Useful.Fit.Avg.Verif.Useful.Fit.Avg.
Human4.941.841.322.702.742.962.002.57
Opus 4.83.184.124.243.852.953.803.093.28
GPT-5.6 Sol3.963.714.804.163.814.144.774.24

Table R1: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol without reference answers or code execution.

BenchmarkVerif.Useful.Fit.Avg.Verif.Useful.Fit.Avg.
Human4.881.751.502.712.973.132.352.82
Opus 4.83.144.404.684.073.444.084.113.88
GPT-5.6 Sol4.224.084.984.434.304.404.964.55

Table R2: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol without reference answers, with code execution enabled.

BenchmarkVerif.Useful.Fit.Avg.Verif.Useful.Fit.Avg.
Human4.861.821.452.713.323.092.382.93
Opus 4.83.324.364.784.153.184.183.903.75
GPT-5.6 Sol4.344.264.984.534.384.364.944.56

Table R3: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol with reference answers and code execution enabled.


Due to the character limit, the response is continued below.

AuthorsOpenReview ↗

Initial Response by Authors (2/2)

Comment 2: Independent retesting of initial and selected tasks is not reported. How were tasks that stopped early included in the curves? The gains may therefore partly reflect selection of unlucky solver runs.

Response 2: Thank you for your constructive comment. In the original refinement curves, the last difficulty ratings of tasks that stopped early after being judged hard and verifiable were carried forward to subsequent rounds.

We further re-evaluate 10 materials-science tasks generated by Claude across refinement rounds. For tasks that stopped early, this re-evaluation uses the unchanged QA with three independent solver trials at each subsequent round. As shown in Table R4, the solving ratio decreases, while the proportion of hard tasks increases, suggesting that the difficulty gains from iterative refinement are also observed under independent re-testing. In response to your comment, we have added these independent re-evaluation results as Fig. 10 in the revised paper.

Refinement RoundSolving Ratio (%)Hard Tasks (%)
0100.00.0
1100.00.0
283.310.0
380.010.0
470.020.0
580.020.0
660.030.0
756.730.0
863.330.0
963.330.0

Table R4: Independent re-evaluation of Claude-generated materials-science tasks across refinement rounds.


Comment 3: What kinds of independent checks were performed on the final reference answers?

Response 3: The refinement procedure leverages verifiability judgement to check whether the question and provided data support a unique, well-defined answer. Specifically, the judge reviews the reference answer together with the question, data, and solver trajectories, assessing whether the target is sufficiently specified and whether disagreements arise from task ambiguity or incorrect solver analyses. When the judge identifies ambiguity or inconsistencies, its feedback guides revisions to the recipe or concept, followed by task re-generation via concept or recipe revision. For the final benchmark, we select only tasks judged verifiable.



We thank the reviewer for your time and feedback and hope that our responses have addressed your concerns. If you have any further questions, we are happy to address them.

AuthorsOpenReview ↗

Brief Summarization of Rebuttal for Reminder

Dear Reviewer xDWE,

We sincerely appreciate your time and constructive feedback. To facilitate your review, we briefly summarize our rebuttal below.


C1-1. Materials-science verifiability

We analyze representative cases to understand the verifiability gap. The high verifiability of human-curated materials-science tasks mainly reflects directly checkable answers, even when these tasks require little scientific analysis.


C1-2. Reliability of quality assessment

To strengthen verification, we assessed materials-science and clinical-imaging tasks with reference-answer access and code execution. Generated benchmarks received higher average quality scores than human-curated benchmarks under these more rigorous assessment settings.


C2. Effectiveness of iterative refinement: Independent retesting of initial and selected tasks

To address the possibility of unlucky solver runs, we re-evaluated materials-science tasks across refinement rounds using fresh, independent trials. These results support the effectiveness of our refinement procedure in increasing task difficulty even under independent evaluation.



We sincerely thank you again for your constructive feedback and hope that the additional results and clarifications have helped address your concerns. As the discussion period approaches its end, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your main concerns.

Best regards,

Authors

Reviewer_czH4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AutoSciBench uses concept and recipe planning to build scientific questions, data files, and reference answers. Solver feedback refines these tasks, and lessons from earlier tasks guide later generation. Generated tasks receive higher average LLM quality ratings across three domains; solver accuracy is generally lower than on the selected human benchmarks in computational biology and materials science.

Critical strengths

Generated materials-science tasks separate solver scores more clearly than the human benchmark, where all solvers score 92.7–97.7% (Table 1).

Critical weaknesses
  1. Reference answers lack independent verification. Builders and refinement judges check validity, but final quality judges cannot run code or see reference answers (A.6–A.7; Fig.21). Opus-generated tasks have both lower solver accuracy and lower verifiability scores than GPT-generated tasks (Tables 1–2). The lower accuracy may partly reflect ambiguous questions or incorrect answers; the paper does not report how often these occur.

  2. Figs.4/6 show difficulty versus refinement round, but not task quality. Lower solver success could reflect more ambiguous questions or incorrect answers, rather than harder tasks. Repeated generation followed by selecting the hardest candidates could also lower success without feedback. The paper lacks a no-feedback baseline with the same budget and stopping/selection rules (A.7–A.8; §4.1).

Questions for authors

Questions

  1. When no task is both hard and verifiable within the round limit, which candidate is selected? Report the final task counts by verifiability label (§4.1; A.8).

Minor comments

  1. §4.3's claim that increased difficulty does not reduce measured quality is too broad. Materials-science verifiability falls from 4.76 for human tasks to 3.49 for Opus-generated tasks and 3.93 for GPT-generated tasks (Table 2).

  2. The abstract should state that the 22.4- and 25.5-percentage-point reductions apply specifically to Opus-generated tasks.

  3. The 22.4-point CompBio gap is not measured under fully matched reasoning settings. Six of ten human leaderboard scores use a different or unreported reasoning effort, whereas generated tasks use high effort (Table 1; A.10).

  4. Table 8's “Benchmark Difficulty” column reports solver accuracy. Label it accordingly and state that lower means harder.

  5. State the denominators and how stopped runs and shared initial tasks enter Figs.4/6 and Table 3. The reported 15.0% cannot represent an integer count out of 50 tasks.

8 comments
AuthorsOpenReview ↗

Initial Response by Authors (1/3)

We thank the reviewer for the constructive comments. We initially address your concerns below.



Comment 1: The final quality judges cannot run code or see reference answers.

Response 1: We appreciate your constructive comment. To ensure a fair comparison with human-curated benchmarks whose reference answers are not publicly available, such as CompBioBench, we withheld reference answers from all judges, while using read-only inspection tools to focus the assessment on the quality of the questions and supplied data rather than on the judges’ ability to solve the tasks. Nevertheless, we additionally evaluate benchmark quality with code execution enabled, both with and without reference answers. As shown in Tables R1, 2, and 3, both generated benchmarks consistently receive higher average quality scores than the human-curated benchmarks, indicating that the higher average quality scores persists when judges can execute code and access reference answers.

BenchmarkVerif.Useful.Fit.Avg.Verif.Useful.Fit.Avg.
Human4.941.841.322.702.742.962.002.57
Opus 4.83.184.124.243.852.953.803.093.28
GPT-5.6 Sol3.963.714.804.163.814.144.774.24

Table R1: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol without reference answers or code execution.

BenchmarkVerif.Useful.Fit.Avg.Verif.Useful.Fit.Avg.
Human4.881.751.502.712.973.132.352.82
Opus 4.83.144.404.684.073.444.084.113.88
GPT-5.6 Sol4.224.084.984.434.304.404.964.55

Table R2: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol without reference answers, with code execution enabled.

BenchmarkVerif.Useful.Fit.Avg.Verif.Useful.Fit.Avg.
Human4.861.821.452.713.323.092.382.93
Opus 4.83.324.364.784.153.184.183.903.75
GPT-5.6 Sol4.344.264.984.534.384.364.944.56

Table R3: Benchmark quality in materials science (left) and clinical imaging (right), assessed by GPT-5.6 Sol with reference answers and code execution enabled.


Comment 2: The lower accuracy may partly reflect ambiguous questions or incorrect answers.

Response 2: We appreciate your insightful feedback. To see the relationship between task verifiability and solver performance, we plot solving ratios aggregated across solvers against the mean verifiability scores from the two judges in Fig. 6 of the revised paper. The low R2R^2 value of 0.074 indicates a weak correlation between the verifiability and accuracy, with substantial variation in solving ratios even among tasks receiving high verifiability scores.

To further analyze the relationship between task verifiability and solver performance, we compare Claude- and GPT-generated tasks across verifiability score ranges. As shown in Table R4, higher verifiability scores do not consistently correspond to higher mean solving ratios, consistent with the weak linear relationship observed in Fig. 6. Moreover, Claude-generated tasks show lower solving ratios within every score range, indicating that the Claude-generated tasks remain more challenging for the evaluated solvers at comparable levels of verifiability.

Verifiability Score (s)Claude-generated Solving RatioGPT-generated Solving Ratio
1.0≤s<2.01.0 \leq s < 2.062.1%72.0%
2.0≤s<3.02.0 \leq s < 3.057.7%68.5%
3.0≤s<4.03.0 \leq s < 4.063.6%68.3%
4.0≤s≤5.04.0 \leq s \leq 5.074.8%78.5%

Table R4: Mean solving ratios of Claude- and GPT-generated tasks by verifiability score range.


Due to the character limit, the response is continued below.

AuthorsOpenReview ↗

Initial Response by Authors (2/3)

Comment 3: Repeated generation followed by selecting the hardest candidates could also lower success without feedback. The paper lacks a no-feedback baseline with the same budget and stopping/selection rules.

Response 3: We additionally evaluate a no-feedback baseline with more generation attempts than AutoSciBench. The baseline (no-feedback) which does not perform the refinement obtains 300 independent tasks, whereas AutoSciBench uses 255 attempts to construct 30 final tasks. We then select the 30 baseline tasks with the lowest solving ratios, matching the final output size. As shown in Table R5, AutoSciBench produces more hard tasks despite using fewer attempts compared to no-feedback baseline, suggesting that feedback-guided refinement is crucial to the task difficulty.

MethodEvaluated tasksSolving Ratio (%) ↓Hard Proportion (%) ↑
AutoSciBench3047.7836.67
no-feedback30099.220.33
no-feedback + hardest-task selection3092.223.33

Table R5: Comparison with independent task generation and hardest-task selection in materials science using Claude Opus 4.8.


Comment 4: When no task is both hard and verifiable within the round limit, which candidate is selected?

Response 4: We first prioritize verifiability and then select the task with the highest difficulty among verifiable candidates across refinement rounds. Notably, for each concept included in the final benchmarks, at least one task was verifiable, allowing us to select a verifiable task even when no hard candidate was available.


Comment 5: §4.3's claim that increased difficulty does not reduce measured quality is too broad. Materials-science verifiability falls from 4.76 for human tasks to 3.49 for Opus-generated tasks and 3.93 for GPT-generated tasks (Table 2).

Response 5: We agree that our original statement was too broad. We have revised Section 4.3 to explicitly acknowledge verifiability in materials science as an exception.

On the other hand, we would like to note that the high verifiability of human-curated materials-science tasks is due to their directly checkable answers. As illustrated in Fig. 8 of Appendix B.1, a code-completion question can be verified directly against the provided source code, receiving high verifiability scores despite requiring little scientific analysis.


Comment 6: The 22.4-point CompBio gap is not measured under fully matched reasoning settings. Six of ten human leaderboard scores use a different or unreported reasoning effort, whereas generated tasks use high effort (Table 1; A.10).

Response 6: Unfortunately, the reference answers for CompBioBench are not publicly available. Therefore, we use the official leaderboard scores.


Due to the character limit, the response is continued below.

AuthorsOpenReview ↗

Initial Response by Authors (3/3)

Comment 7: State the denominators and how stopped runs and shared initial tasks enter Figs.4/6 and Table 3. The reported 15.0% cannot represent an integer count out of 50 tasks.

Response 7: We apologize for the lack of clarity in the reported denominators and aggregation procedure. For Table 3, the two conditions share the initial 10 tasks, and the remaining 40 tasks are generated with or without experience. We exclude the shared 10 tasks from the reported results. Thus, the denominator for the reported 15.0% is 40, corresponding to the non-shared tasks.

For Figs. 4/9 of the revised paper (previously Figs. 4/6), we compute the proportions by pooling generated tasks across the three domains, with 300 and 150 tasks in Fig. 4 and 6, respectively. The runs are stopped if the task is judged both hard and verifiable, or if the refinement round limit is reached.

We have revised the paper as follows:

  • Section 4.4: “pooling tasks generated by Claude-Opus-4.8 and GPT-5.6-Sol” →\rightarrow “pooling all 300 tasks generated by Claude-Opus-4.8 and GPT-5.6-Sol across the three domains”.
  • Section B.2, Figure 9 caption: “We report proportions by gathering tasks across all three benchmarks.” →\rightarrow “We report proportions by gathering tasks across all three benchmarks, with 150 tasks per generator.”
  • Section 4.5: “Both share 10 initial concepts and generate 50 tasks in total, retaining iterative task refinement.” →\rightarrow “Both conditions share 10 initial tasks and generate 40 additional tasks per condition, retaining iterative task refinement. We exclude the shared initial tasks and report results for the 40 non-shared tasks in each condition in Table 3.”
  • Section A.8: “We repeat this process for up to 10 refinement rounds.” →\rightarrow “We repeat this process for up to 10 refinement rounds, stopping early once the task is judged both hard and verifiable.”

Comment 8: The abstract should state that the 22.4- and 25.5-percentage-point reductions apply specifically to Opus-generated tasks. Table 8's “Benchmark Difficulty” column reports solver accuracy. Label it accordingly and state that lower means harder.

Response 8: Thank you for pointing these out. We have revised the paper as follows:

  • Abstract: “Generated benchmarks” →\rightarrow “Benchmarks generated by Claude Opus 4.8”.
  • Section B.6: “Benchmark Difficulty” →\rightarrow “Avg. Solver Accuracy”.


We thank the reviewer for your time and feedback and hope that our responses have addressed your concerns. If you have any further questions, we are happy to address them.

Reviewer_FvswOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces AutoSciBench, an automated framework for constructing and refining benchmarks for scientific agents. The framework decomposes task generation into two levels: a high-level concept specifying the scientific domain, data modality, and required analytical operations, and a low-level recipe specifying how to construct the question, associated scientific data, and reference answer.

Generated tasks are evaluated through multiple solver attempts and LLM-based judging. Solver feedback guides iterative refinement of the recipe or concept to improve verifiability and difficulty. The framework additionally distills refinement trajectories into reusable experience cards that inform subsequent concept generation.

Critical strengths
  1. The motivation is clear and it's interesting to automatically generate benchmarks for agents.
  2. The structured, feedback-driven refinement is reasonable and has encouraging empirical results.
Critical weaknesses
  1. My primary concern is whether the generated tasks are actually reliable and scientifically meaningful, rather than merely difficult for the evaluated agents. There are also concerns about the scientific relevance of some generated tasks. For example, the cardiothoracic-ratio task explicitly requires calculating the answer from provided JSON landmark measurements rather than interpreting the accompanying radiograph. I have questions about whether all generated tasks assess the intended scientific capabilities and realistic research workflows. Moreover, Table 2 shows that generated materials-science tasks actually receive lower verifiability scores than the human-curated tasks, despite their higher aggregate quality scores.

  2. Difficulty improvements are potentially confounded by adaptive selection, and the experimental comparisons do not sufficiently isolate the framework's contribution.

  3. The main components, including hierarchical benchmark planning, solver-feedback-driven difficulty optimization, and experience-guided generation, have substantial precedents in prior work.

Questions for authors
  1. Are the solver evaluations in Table 1 performed using fresh, independent runs after the final tasks have been selected, or do they reuse generation-time solver results?

  2. How does the complete framework compare against simpler alternatives such as direct scientific task generation followed by solver-feedback refinement, or rejection sampling over a larger set of independently generated tasks?

6 comments
AuthorsOpenReview ↗

Initial Response by Authors (1/2)

We sincerely thank you for your constructive comments and positive feedback:

  • Clear motivation for automatic benchmark generation
  • Structured, feedback-driven refinement is reasonable
  • Encouraging empirical results

We initially address your concerns below.



Comment 1-1: Are the generated tasks scientifically meaningful, rather than merely difficult for the evaluated agents?

Response 1-1: To evaluate scientific meaningfulness, we leverage usefulness scores as in Table 2. This score is designed to assess whether a task evaluates scientifically meaningful capabilities relevant to real research workflows. Averaged across the three domains, Claude- and GPT-generated tasks receive scores of 4.01 and 3.77, respectively, on a five-point scale where 3 denotes “acceptable”, suggesting that the generated tasks are generally scientifically relevant, although individual tasks may assess a limited part of a scientific workflow.


Comment 1-2: There are also concerns about the scientific relevance of some generated tasks. For example, the cardiothoracic-ratio task explicitly requires calculating the answer from provided JSON landmark measurements rather than interpreting the accompanying radiograph.

We agree that generated benchmarks can include tasks that assess basic capabilities but require limited scientific analysis. However, we would like to note that this scientific relevance is correctly assessed by the usefulness score. The cardiothoracic-ratio example does not require direct interpretation of the radiograph. Both judges assign it a poor usefulness score of 2/5, recognizing the clinical relevance of CTR measurement while noting that providing anatomical landmarks removes the need to extract measurements from the image. The judges’ rationales are below:

  • Claude Opus 4.8: Cardiothoracic ratio is a genuinely meaningful clinical measure of cardiomegaly, but here reduces to arithmetic on given coordinates rather than measurement extraction.
  • GPT 5.6: Cardiothoracic-ratio measurement is clinically meaningful, but the task reduces an imaging workflow to selecting extrema from precomputed landmarks rather than assessing anatomy or annotation quality.

On the other hand, we also would like to note that some human-curated tasks could scientifically be less meaningful. As shown in Fig. 7 of the revised paper, a MedCTA tissue-identification task is answerable by reading the “Breast Carcinoma” label in the image and both judges assign it a usefulness score of 2/5. Even though this example is imaging-relevant, it lacks clinical relevance.

Therefore, our usefulness assessment captures limitations in both generated and human-curated tasks, while the benchmark-wide scores in Table 2 support the overall scientific relevance of the generated benchmarks.


Comment 1-3: Table 2 shows that generated materials-science tasks actually receive lower verifiability scores than the human-curated tasks, despite their higher aggregate quality scores.

Response 1-3: The high verifiability scores of human-curated tasks in materials-science are caused by directly checkable targets. Specifically, the tasks are the code completion with multiple choice, which are often easily answerable as shown in the example in Fig. 8 of the revised paper. Whereas its verifiability score is high (5), the usefulness and fitness scores are significantly low (1) due to the simplicity of the question. On the other hand, human-curated tasks in other domains and generated tasks require the computation and interpretation of the desired estimand, which are less straightforward, but meaningful and fit to evaluate scientific agents.


Comment 2: Difficulty improvements are potentially confounded by adaptive selection, and the experimental comparisons do not sufficiently isolate the framework's contribution.

Response 2: We appreciate your insightful comment. To rigorously test the difficulty improvements, we report the re-tested solving ratio and proportion of hard tasks over refinement rounds for 10 Claude-generated tasks in the material science. As shown in Table R4, the proportion of hard tasks increases, while the solving ratio decreases overall, indicating that the difficulty improvements are not confined to the solver trials used for adaptive selection and are also observed under independent re-evaluation. In response to your comment, we have added these independent re-evaluation results as Fig. 10 in the revised paper.

Refinement RoundSolving Ratio (%)Hard Tasks (%)
0100.00.0
1100.00.0
283.310.0
380.010.0
470.020.0
580.020.0
660.030.0
756.730.0
863.330.0
963.330.0

Table R1: Independent re-evaluation of Claude-generated materials-science tasks across refinement rounds.


Due to the character limit, the response is continued below.

AuthorsOpenReview ↗

Initial Response by Authors (2/2)

Comment 3: The main components, including hierarchical benchmark planning, solver-feedback-driven difficulty optimization, and experience-guided generation, have substantial precedents in prior work.

Response 3: Our main contribution is a framework tailored to scientific-agent evaluation. We agree that planning, feedback-based optimization, and experience-guided generation are established techniques. However, our design adapts them to two specific evaluation requirements.

First, information acquisition through interaction requires specifying both the environment and the analytical operations to be evaluated. To address this requirement, we devise concepts which define the scientific domain, data modality, and required operations, and recipes that determine which information is provided in the question and which should be recovered through interaction with the data and tools.

Second, scientific reasoning from indirect evidence requires interpreting observations that do not directly indicate whether an analysis is correct. To address this challenge, we aim to modify the analytical requirements and data design to better assess raw-data re-examination, interpretation, and evidence integration, and propose an iterative refinement framework that revises concepts and recipes based on solver feedback.

Accordingly, our contribution within Applications to Sciences addresses the practical challenge of automatically constructing and updating scientific-agent benchmarks around these requirements, supporting evaluation that can keep pace with advancing agents.


Comment 4: Are the solver evaluations in Table 1 performed using fresh, independent runs after the final tasks have been selected, or do they reuse generation-time solver results?

Response 4: The solver evaluation in Table 1 was performed with independent runs after the final task selection for a fair comparison.


Comment 5: How does the complete framework compare against simpler alternatives such as direct scientific task generation followed by solver-feedback refinement, or rejection sampling over a larger set of independently generated tasks?

Response 5: As shown in Fig. 5 (w/o C+R), we compared a simpler alternative that directly generates scientific tasks and refines them with solver-feedback, not leveraging the intermediate plans such as concept and recipe. In other words, this variant does not explicitly consider agentic capabilities and scientific reasoning, directly revising the questions and data. As a result, it underperforms when using both or only concepts, indicating that explicit intermediate plans (i.e., concept and recipe) help translate solver feedback into more effective revisions of the analytical requirements and data design, leading to more challenging scientific tasks.

We further evaluate independent task generation without refinement, while retaining intermediate plans such as concepts and recipes. For this comparison, we consider 30 of the 50 materials-science tasks generated by AutoSciBench and independently generate a larger pool of 300 tasks without refinement, matching the number of task generation. As shown in Table R2, AutoSciBench generates more hard tasks, even compared with selecting the hardest independently generated tasks. These results suggest that feedback-guided iterative refinement plays a key role in increasing task difficulty beyond repeated generation and selection alone.

MethodEvaluated QAsSolving Ratio (%) ↓Hard Proportion (%) ↑
AutoSciBench3047.7836.67
w/o refinement30099.220.33
w/o refinement + hardest-task selection3092.223.33

Table R2: Comparison with independent task generation and hardest-task selection in materials science using Claude Opus 4.8.



We thank the reviewer for your time and feedback and hope that our responses have addressed your concerns. If you have any further questions, we are happy to address them.

AuthorsOpenReview ↗

Brief Summarization of Rebuttal for Reminder

Dear Reviewer Fvsw,

We sincerely appreciate your time and constructive feedback. To facilitate your review, we briefly summarize our rebuttal below.



C1. Scientific meaningfulness of generated tasks

To assess scientific meaningfulness, we analyzed representative cases and overall usefulness scores, supporting the overall scientific relevance of generated tasks: the usefulness assessment identifies tasks with limited scientific value, while generated tasks achieve higher overall usefulness than human-curated tasks. We also clarified that the high verifiability of human-curated materials-science tasks mainly reflects directly checkable answers.


C2 & C4. Validity of difficulty gains

To test whether difficulty gains are an artifact of adaptive selection, we independently re-evaluated materials-science tasks across refinement rounds and showed that difficulty still increased overall. These results support the effectiveness of our refinement procedure in increasing task difficulty even under independent evaluation. We also clarified that the reported performance in Table 1 is from independent runs after the final task selection.


C3. Contribution beyond prior work

We clarified that our work addresses the challenge of translating desired scientific capabilities, including information acquisition through interaction and scientific reasoning from indirect evidence, into concrete, verifiable evaluation tasks. Our main contribution, within our primary area of Applications to Sciences, is a framework for automatically constructing and improving scientific-agent evaluations in response to agents’ capabilities and failure modes.


C5. Comparison with simpler alternatives

To isolate the roles of planning and refinement, we compared against direct generation with refinement and independent generation with rejection sampling. The results show that AutoSciBench produces more challenging tasks than both alternatives, supporting the effectiveness of explicit planning and feedback-guided refinement.



We sincerely thank you again for your constructive feedback and hope that the additional results and clarifications have helped address your concerns. As the discussion period approaches its end, we would greatly appreciate it if you could let us know whether our responses have sufficiently addressed your main concerns.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.