- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SCIARP as a benchmark to add scientific perturbation and examine the performance of the science agent. It aims to see how perturbation affects the agents in multi-turn problem solving and how agents resist to perturbation. Results show that current agents are not robust enough under perturbation.
Critical strengths
- The motivation is reasonable. Scientific accuracy is indeed crucial in multi-turn tasks. Errors in earlier rounds can impact later rounds.
- The experimental results show that perturbations significantly affect model performance, indicating quantitatively for the problem.
Critical weaknesses
- The data amount is relatively small. More data could be added since this is a benchmark.
- Although the generated perturbations were manually reviewed, whether the human verifier possessed the expertise to assess their reasonableness and whether they met expectations is an important issue, as these scientific questions and interactions have a certain level of knowledge requirements to review.
- Are these perturbations reasonable interferences? Are they realistically possible, or deliberately introduced errors unlikely to occur in reality to undermine the model's reasoning?
- The necessity of some perturbations is questionable. U2 Dimension/unit and U3 Variable/symbol are examples. In practice, the probability of the model actually making mistakes at these two levels is relatively small. The motivation used here as one of the perturbations is not strong enough. I hope the authors to add clearer motivational explanations to all perturbations and to have statistics on the model making these mistakes in the absence of perturbations, which would be more realistic.
Questions for authors
Please address the weaknesses above. Figure 1: The text is too small, making it hard to read.
7 comments
esponse to Reviewer ruSN
Weakness 3: Realism and Plausibility of Perturbations
Response: We appreciate the reviewer’s helpful comment. SciARP is designed to evaluate agents against scientifically plausible imperfections that may arise during normal scientific problem solving, rather than artificial attacks tailored to undermine specific models. Although our perturbations are intentionally constructed for controlled evaluation, their underlying mechanisms are grounded in documented issues in scientific research and communication.
First, all 13 perturbation types are motivated by phenomena documented in existing scientific literature. For example, unit inconsistencies have caused real engineering failures (NIST), missing experimental details have been documented in published enzyme-function studies (Halling et al., 2018), and citation inaccuracies have been empirically reported in medical research (Mogull, 2017). Similarly, inconsistencies between reported statistical results (García-Berthou & Alcaraz, 2004), overreaching causal claims in observational research (Isch et al., 2026), and the influence of authority cues on credibility judgments (Hoogeveen et al., 2022) have all been documented. These findings establish that the underlying phenomena represented by our taxonomy are not hypothetical errors invented solely for evaluation. Detailed motivations and supporting literature for all 13 types are provided in Appendix B of the revised paper.
Second, these imperfections can reach scientific agents through their normal workflows. Agents retrieve literature, analyze datasets, invoke tools, follow task specifications, and build on intermediate results from previous interactions. These channels may expose agents to inaccurate citations, inconsistent statistics, conflicting findings, different units or notations, missing conditions, and misleading intermediate reasoning. SciARP translates these documented imperfections into controlled, task-conditioned perturbations grounded in the original scientific problems and multi-turn interaction contexts. The perturbations follow predefined mechanisms, are not tailored to any specific evaluated model, and are fixed before evaluation and applied consistently across all models. Importantly, our goal is to evaluate the potential impact of these plausible imperfections, not to estimate their natural occurrence frequency in unperturbed agent workflows.
Third, every generated perturbation undergoes LLM-based and human dual validation. The validators examine its conformity to the intended perturbation mechanism, scientific plausibility, preservation of the original scientific objective, and cross-turn consistency. Only instances passing both validation stages are retained; instances introducing unintended scientific changes are excluded.
In the revised paper, we have added detailed literature-supported motivations and representative examples for all 13 perturbation types in Appendix B (Pages 18–21), and clarified the perturbation applicability, instantiation, and dual-validation procedures in Appendix C (Pages 21–22).
Response to Reviewer ruSN
Weakness 4: Motivation and Necessity of Perturbation Types
Response: We thank the reviewer for this comment. We further clarify the motivation of all 13 perturbation types and the necessity of U2 and U3 in evaluating multi-turn scientific reasoning.
First, the 13 perturbation types cover complementary capabilities across four stages of scientific problem solving. Specifically, U1–U4 examine whether agents maintain correct problem representations and specifications; E1–E4 assess their ability to evaluate scientific premises, citations, and evidence; R1–R3 examine the consistency and validity of intermediate reasoning; and C1–C2 assess whether agents maintain calibrated uncertainty and evidence-grounded conclusions. These types represent different failure mechanisms rather than redundant variations of the same error. Their individual motivations and supporting literature are detailed in Appendix B of the revised paper. The literature evidence establishes the relevance of the underlying phenomena, while our controlled experiments investigate their potential impact on multi-turn scientific problem solving.
Second, U2 and U3 target cross-step consistency in scientific reasoning, rather than isolated unit conversion or symbol recognition. Scientific agents may encounter quantities expressed in different units and scales, as well as context-dependent variables and symbols, when integrating information from different sources, datasets, and tools. Unit inconsistencies have caused real engineering failures, such as the loss of NASA's Mars Climate Orbiter (NIST), while the reuse of identical mathematical symbols with different meanings is documented in scientific literature (Asakura et al., 2022). U2 examines whether agents maintain dimensional and quantitative consistency when differently represented quantities are reused across dependent calculations. U3 examines whether agents correctly resolve context-dependent variable meanings and maintain consistent symbolic references throughout multi-step reasoning. Although individual unit conversions or symbol interpretations may be straightforward, inconsistencies in these representations can propagate through subsequent reasoning and alter final scientific conclusions.
Third, our experimental results demonstrate that these perturbations can compromise multi-turn scientific reasoning. As reported in Tables 7–8, U2 and U3 cause average TACC degradation of 5.52 and 5.58 percentage points, respectively. They also degrade IC by 2.03 and 2.19 percentage points and RV by 3.84 and 4.14 percentage points, despite relatively small reductions in TP (0.60 and 0.43 percentage points). Figure 3 further shows that their effects persist to the final turn in 62% and 66% of cases, respectively. Representative trajectories in Figures 5–6 illustrate how unit-conversion and symbol-binding errors propagate into subsequent calculations and final results.
In the revised paper, we have expanded the motivations and supporting literature for all 13 perturbation types in Appendix B (Pages 18–21), with representative examples and detailed definitions.
Question 1: Addressing Reviewer Concerns and Improving Figure Readability
Response: We appreciate the reviewer’s constructive feedback. We have addressed the concerns regarding dataset scale, human validation, perturbation realism, and the motivation of individual perturbation types through the revisions and additional analyses described above. We have also revised Figure 1 by increasing the font size, simplifying textual content, and adjusting the layout to improve readability. The updated figure reflects the expanded benchmark of 720 scientific tasks from five sources.
We would be happy to provide further clarification and address any additional questions the reviewer may have.
Response to Reviewer ruSN
We sincerely thank the reviewer for the thoughtful evaluation and constructive feedback. We appreciate the recognition of the importance of studying scientific-agent robustness in multi-turn problem solving and the empirical value of our findings in demonstrating the impact of scientific perturbations. In response, we have expanded SciARP from 620 to 720 scientific tasks, clarified the expertise of human reviewers and strengthened the LLM–human dual-validation procedures, provided literature-based motivations for all 13 perturbation types, and further explained the necessity of U2 and U3. We have also revised Figure 1 to improve its readability. All modifications in the revised manuscript are highlighted in blue.
Weakness 1: Dataset Scale and Coverage
Response: We appreciate the reviewer’s helpful comment. To address the concern about dataset scale and broaden source coverage, we have expanded SciARP from 620 to 720 scientific problems by incorporating 100 additional physics reasoning problems from the recently published PhysLogic benchmark. This addition introduces a fifth data source and further broadens the benchmark's coverage of scientific reasoning.
The 100 additional problems undergo the same multi-turn task construction and LLM–human dual-validation pipeline as the original data, resulting in 100 additional interdependent multi-turn tasks. We then rerun the complete evaluation across all eight models and 13 perturbation types on the expanded 720-task benchmark under the same experimental protocol.
In the revised paper, we have updated Figure 1 (Page 5), clarified the five data sources and task construction in Section 4.2 (Pages 5–6), and provided the source selection and quality-control procedures in Appendix A (Pages 16–17). All corresponding experimental results evaluated by integrating the 100 additional problems have been updated.** All modifications in the revised paper are highlighted in blue.
Weakness 2: Expertise and Reliability of Human Validation
Response: We appreciate the reviewer’s helpful comment. We clarify the expertise of our human reviewers and the quality-control procedures used in benchmark construction.
First, SciARP is grounded in established scientific benchmarks with verified reference solutions. The original problems have independently verifiable correctness and require non-trivial multi-step reasoning, providing explicit scientific references for validating the constructed tasks and perturbations.
Second, both human reviewers involved in task construction and perturbation validation have at least five years of scientific research experience. They assess scientific correctness and reasoning validity against the original problems and verified reference solutions, following predefined criteria for task construction and perturbation validation.
Third, human validation follows explicit, predefined criteria rather than relying solely on subjective judgments. Reviewers assess constructed tasks against the original scientific problems and verified reference solutions. For task construction, they examine scientific correctness, reasoning completeness, consistency with the original problem and reference solution, and cross-turn dependency validity. For perturbation construction, they further assess conformity to the intended perturbation type, scientific plausibility, targeted modification, preservation of the original scientific objective, and the absence of unintended scientific changes.
Fourth, we assess inter-reviewer reliability and adopt dual validation for quality control. The two human reviewers independently evaluate the constructed tasks before resolving disagreements through consensus adjudication. In addition, we combine LLM-based and human validation, applying the same predefined criteria to both. Only instances passing both validation stages are retained; any instance failing either stage is excluded from the benchmark.
In the revised paper, we have clarified the human reviewers’ research experience and quality-control procedures for multi-turn task construction in Appendix A (Pages 16–17) and perturbation validation in Appendix C (Page 22).