acceptodds
Desk rejected at ICLR 2027

The Marathon of Scientific Reasoning: Robustness of Scientific Agents to Perturbations in Multi-Turn Interaction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM)-based scientific agents are increasingly used for scientific problem solving, yet their robustness to imperfections arising during multi-turn interactions remains poorly understood. We introduce SciARP (Scientific Agent Robustness to Perturbations), a benchmark for evaluating scientific agents under scientifically plausible perturbations throughout multi-turn problem solving. SciARP transforms 620 scientific problems into interdependent tasks of 3–13 turns and defines 13 perturbation types spanning problem understanding, evidence processing, reasoning, and conclusion formation. Clean and perturbed versions of each task are independently executed under matched settings, producing paired live trajectories for evaluating both task success and process reliability. Experiments across eight LLMs from four model families reveal three key robustness characteristics. First, different classes of scientific perturbations exhibit distinct robustness profiles and can decouple task progression from scientific reliability: agents may continue advancing through the task even after their information or reasoning has become unreliable. Second, stronger clean-task performance does not necessarily translate into stronger robustness, as models with higher clean-task accuracy can exhibit larger degradation under perturbation. Third, perturbation effects exhibit strong temporal dynamics: they may remain latent for multiple turns before emerging and subsequently propagate through downstream dependencies. Together, these findings show that current scientific agents remain insufficiently robust to scientifically plausible perturbations, with failures often remaining undetected, propagating, and resisting recovery.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, AI for Science, Multi-Turn Reasoning, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ruSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCIARP as a benchmark to add scientific perturbation and examine the performance of the science agent. It aims to see how perturbation affects the agents in multi-turn problem solving and how agents resist to perturbation. Results show that current agents are not robust enough under perturbation.

Critical strengths
  1. The motivation is reasonable. Scientific accuracy is indeed crucial in multi-turn tasks. Errors in earlier rounds can impact later rounds.
  2. The experimental results show that perturbations significantly affect model performance, indicating quantitatively for the problem.
Critical weaknesses
  1. The data amount is relatively small. More data could be added since this is a benchmark.
  2. Although the generated perturbations were manually reviewed, whether the human verifier possessed the expertise to assess their reasonableness and whether they met expectations is an important issue, as these scientific questions and interactions have a certain level of knowledge requirements to review.
  3. Are these perturbations reasonable interferences? Are they realistically possible, or deliberately introduced errors unlikely to occur in reality to undermine the model's reasoning?
  4. The necessity of some perturbations is questionable. U2 Dimension/unit and U3 Variable/symbol are examples. In practice, the probability of the model actually making mistakes at these two levels is relatively small. The motivation used here as one of the perturbations is not strong enough. I hope the authors to add clearer motivational explanations to all perturbations and to have statistics on the model making these mistakes in the absence of perturbations, which would be more realistic.
Questions for authors

Please address the weaknesses above. Figure 1: The text is too small, making it hard to read.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ruSN

We sincerely thank the reviewer for the thoughtful evaluation and constructive feedback. We appreciate the recognition of the importance of studying scientific-agent robustness in multi-turn problem solving and the empirical value of our findings in demonstrating the impact of scientific perturbations. In response, we have expanded SciARP from 620 to 720 scientific tasks, clarified the expertise of human reviewers and strengthened the LLM–human dual-validation procedures, provided literature-based motivations for all 13 perturbation types, and further explained the necessity of U2 and U3. We have also revised Figure 1 to improve its readability. All modifications in the revised manuscript are highlighted in blue.

Weakness 1: Dataset Scale and Coverage

Response: We appreciate the reviewer’s helpful comment. To address the concern about dataset scale and broaden source coverage, we have expanded SciARP from 620 to 720 scientific problems by incorporating 100 additional physics reasoning problems from the recently published PhysLogic benchmark. This addition introduces a fifth data source and further broadens the benchmark's coverage of scientific reasoning.

The 100 additional problems undergo the same multi-turn task construction and LLM–human dual-validation pipeline as the original data, resulting in 100 additional interdependent multi-turn tasks. We then rerun the complete evaluation across all eight models and 13 perturbation types on the expanded 720-task benchmark under the same experimental protocol.

In the revised paper, we have updated Figure 1 (Page 5), clarified the five data sources and task construction in Section 4.2 (Pages 5–6), and provided the source selection and quality-control procedures in Appendix A (Pages 16–17). All corresponding experimental results evaluated by integrating the 100 additional problems have been updated.** All modifications in the revised paper are highlighted in blue.

Weakness 2: Expertise and Reliability of Human Validation

Response: We appreciate the reviewer’s helpful comment. We clarify the expertise of our human reviewers and the quality-control procedures used in benchmark construction.

First, SciARP is grounded in established scientific benchmarks with verified reference solutions. The original problems have independently verifiable correctness and require non-trivial multi-step reasoning, providing explicit scientific references for validating the constructed tasks and perturbations.

Second, both human reviewers involved in task construction and perturbation validation have at least five years of scientific research experience. They assess scientific correctness and reasoning validity against the original problems and verified reference solutions, following predefined criteria for task construction and perturbation validation.

Third, human validation follows explicit, predefined criteria rather than relying solely on subjective judgments. Reviewers assess constructed tasks against the original scientific problems and verified reference solutions. For task construction, they examine scientific correctness, reasoning completeness, consistency with the original problem and reference solution, and cross-turn dependency validity. For perturbation construction, they further assess conformity to the intended perturbation type, scientific plausibility, targeted modification, preservation of the original scientific objective, and the absence of unintended scientific changes.

Fourth, we assess inter-reviewer reliability and adopt dual validation for quality control. The two human reviewers independently evaluate the constructed tasks before resolving disagreements through consensus adjudication. In addition, we combine LLM-based and human validation, applying the same predefined criteria to both. Only instances passing both validation stages are retained; any instance failing either stage is excluded from the benchmark.

In the revised paper, we have clarified the human reviewers’ research experience and quality-control procedures for multi-turn task construction in Appendix A (Pages 16–17) and perturbation validation in Appendix C (Page 22).

AuthorsOpenReview ↗

esponse to Reviewer ruSN

Weakness 3: Realism and Plausibility of Perturbations

Response: We appreciate the reviewer’s helpful comment. SciARP is designed to evaluate agents against scientifically plausible imperfections that may arise during normal scientific problem solving, rather than artificial attacks tailored to undermine specific models. Although our perturbations are intentionally constructed for controlled evaluation, their underlying mechanisms are grounded in documented issues in scientific research and communication.

First, all 13 perturbation types are motivated by phenomena documented in existing scientific literature. For example, unit inconsistencies have caused real engineering failures (NIST), missing experimental details have been documented in published enzyme-function studies (Halling et al., 2018), and citation inaccuracies have been empirically reported in medical research (Mogull, 2017). Similarly, inconsistencies between reported statistical results (García-Berthou & Alcaraz, 2004), overreaching causal claims in observational research (Isch et al., 2026), and the influence of authority cues on credibility judgments (Hoogeveen et al., 2022) have all been documented. These findings establish that the underlying phenomena represented by our taxonomy are not hypothetical errors invented solely for evaluation. Detailed motivations and supporting literature for all 13 types are provided in Appendix B of the revised paper.

Second, these imperfections can reach scientific agents through their normal workflows. Agents retrieve literature, analyze datasets, invoke tools, follow task specifications, and build on intermediate results from previous interactions. These channels may expose agents to inaccurate citations, inconsistent statistics, conflicting findings, different units or notations, missing conditions, and misleading intermediate reasoning. SciARP translates these documented imperfections into controlled, task-conditioned perturbations grounded in the original scientific problems and multi-turn interaction contexts. The perturbations follow predefined mechanisms, are not tailored to any specific evaluated model, and are fixed before evaluation and applied consistently across all models. Importantly, our goal is to evaluate the potential impact of these plausible imperfections, not to estimate their natural occurrence frequency in unperturbed agent workflows.

Third, every generated perturbation undergoes LLM-based and human dual validation. The validators examine its conformity to the intended perturbation mechanism, scientific plausibility, preservation of the original scientific objective, and cross-turn consistency. Only instances passing both validation stages are retained; instances introducing unintended scientific changes are excluded.

In the revised paper, we have added detailed literature-supported motivations and representative examples for all 13 perturbation types in Appendix B (Pages 18–21), and clarified the perturbation applicability, instantiation, and dual-validation procedures in Appendix C (Pages 21–22).

AuthorsOpenReview ↗

Response to Reviewer ruSN

Weakness 4: Motivation and Necessity of Perturbation Types

Response: We thank the reviewer for this comment. We further clarify the motivation of all 13 perturbation types and the necessity of U2 and U3 in evaluating multi-turn scientific reasoning.

First, the 13 perturbation types cover complementary capabilities across four stages of scientific problem solving. Specifically, U1–U4 examine whether agents maintain correct problem representations and specifications; E1–E4 assess their ability to evaluate scientific premises, citations, and evidence; R1–R3 examine the consistency and validity of intermediate reasoning; and C1–C2 assess whether agents maintain calibrated uncertainty and evidence-grounded conclusions. These types represent different failure mechanisms rather than redundant variations of the same error. Their individual motivations and supporting literature are detailed in Appendix B of the revised paper. The literature evidence establishes the relevance of the underlying phenomena, while our controlled experiments investigate their potential impact on multi-turn scientific problem solving.

Second, U2 and U3 target cross-step consistency in scientific reasoning, rather than isolated unit conversion or symbol recognition. Scientific agents may encounter quantities expressed in different units and scales, as well as context-dependent variables and symbols, when integrating information from different sources, datasets, and tools. Unit inconsistencies have caused real engineering failures, such as the loss of NASA's Mars Climate Orbiter (NIST), while the reuse of identical mathematical symbols with different meanings is documented in scientific literature (Asakura et al., 2022). U2 examines whether agents maintain dimensional and quantitative consistency when differently represented quantities are reused across dependent calculations. U3 examines whether agents correctly resolve context-dependent variable meanings and maintain consistent symbolic references throughout multi-step reasoning. Although individual unit conversions or symbol interpretations may be straightforward, inconsistencies in these representations can propagate through subsequent reasoning and alter final scientific conclusions.

Third, our experimental results demonstrate that these perturbations can compromise multi-turn scientific reasoning. As reported in Tables 7–8, U2 and U3 cause average TACC degradation of 5.52 and 5.58 percentage points, respectively. They also degrade IC by 2.03 and 2.19 percentage points and RV by 3.84 and 4.14 percentage points, despite relatively small reductions in TP (0.60 and 0.43 percentage points). Figure 3 further shows that their effects persist to the final turn in 62% and 66% of cases, respectively. Representative trajectories in Figures 5–6 illustrate how unit-conversion and symbol-binding errors propagate into subsequent calculations and final results.

In the revised paper, we have expanded the motivations and supporting literature for all 13 perturbation types in Appendix B (Pages 18–21), with representative examples and detailed definitions.

Question 1: Addressing Reviewer Concerns and Improving Figure Readability

Response: We appreciate the reviewer’s constructive feedback. We have addressed the concerns regarding dataset scale, human validation, perturbation realism, and the motivation of individual perturbation types through the revisions and additional analyses described above. We have also revised Figure 1 by increasing the font size, simplifying textual content, and adjusting the layout to improve readability. The updated figure reflects the expanded benchmark of 720 scientific tasks from five sources.

We would be happy to provide further clarification and address any additional questions the reviewer may have.

Reviewer_BnoSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SCIARP, a benchmark for evaluating the robustness of LLM-based scientific agents. The benchmark examines scientific agents under scientifically plausible perturbations throughout multi-turn problem-solving tasks. It transforms 620 scientific problems into interdependent tasks containing 3-13 turns and defines 13 perturbation types in four categories: problem understanding, evidence processing, reasoning, and conclusion formation. Experiments across eight LLMs from four model families conclude that current scientific agents remain insufficiently robust to scientifically plausible perturbations. Failures often remain undetected, propagate, and resist recovery throughout the reasoning process.

Critical strengths
  • The paper studies a timely and meaningful problem in AI4S applications, where errors might accumulate in the trajectories during reasoning. Therefore, a meaningful answer does not imply that the reasoning process leading to it is correct.
  • The studies of tasks, perturbation types, and stages of problem solving are comprehensive and systematic. Controlled experiments use both original and perturbed tasks to examine the change in trajectories caused by the perturbation.
  • The metrics used are not limited to the final results but also look at the reasoning process. Evaluations are on six types of proprietary models and involve both LLMs and human judges.
  • The conclusions and findings are meaningful and interesting.
Critical weaknesses
  • The ground truth after perturbation might differ, as some perturbed information provided to the agent may lead to a different correct result, or some missing/insufficient necessary information may cause the agent to think in a different way.
  • The agent may ask for further clarification if there are abnormalities (e.g., conflicts, insufficient information). This part is not discussed thoroughly in the paper.
  • The level of perturbation might differ, and the differences in perturbation strengths might cause various impacts on the results and trajectories. The perturbation in different categories might contribute differently.
Questions for authors
  • Are there any statistical significance analyses for the results?
  • How does the evaluation distinguish between genuine reasoning failures and failures due to insufficient/inconsistent information provided?
  • How can the authors ensure the benchmark is constructed to provide a clear comparison between the four perturbation categories? How can they ensure that the questions are represented precisely for the agent's reasoning?
  • Can you provide more details of the human evaluation (e.g., how many trajectories and turns are annotated? what is the disagreement rate between human annotators/between humans and LLMs)
8 comments
AuthorsOpenReview ↗

Response to Reviewer BnoS

We sincerely thank the reviewer for the thoughtful and constructive evaluation of our work. We appreciate the recognition of the importance of scientific-agent robustness in multi-turn problem solving, the systematic design of our four-stage perturbation taxonomy, and the value of our paired clean–perturbed evaluation and process-level metrics. In response to the reviewer’s feedback, we have clarified how the original scientific objective and ground truth are preserved, introduced a per-turn self-reflection and repair mechanism with additional experiments, and added a three-level perturbation strength analysis. We have also expanded the statistical evaluation, conducted failure trajectory attribution, and provided further details on multi-turn task construction and human evaluation. All modifications in the revised manuscript are highlighted in blue.

Weakness 1: Ground-Truth Consistency under Perturbations

Response: We thank the reviewer for this comment. In SciARP, perturbations modify the information or reasoning context encountered by the agent, but do not redefine the original scientific task or establish an alternative ground truth. We ensure this through fixed task references, carefully constructed perturbations, and LLM–human dual validation.

First, our perturbations challenge different aspects of scientific problem solving without establishing alternative ground truths. The 13 perturbation types target different aspects of problem solving: U1–U3 modify information representation, U4 withholds task-relevant conditions, E1–E4 introduce unreliable or conflicting evidence, R1–R3 introduce intermediate reasoning errors or misleading directions, and C1–C2 introduce unsupported contextual pressure. Although these modifications may influence the agent's reasoning and intermediate conclusions, the original problem and y∗y^* remain the fixed references for evaluating their correctness. For example, in E3, a record's statistical classification is changed from Missense to Synonymous in the presented data copy, while the original archived annotation is retained as the benchmark-side reference for validation and evaluation.

Second, reaching y∗y^* under perturbation requires agents to appropriately identify and handle the introduced imperfections, which is precisely what SciARP evaluates. A reliable agent should assess the validity of presented information against scientific knowledge, original evidence, and previously established results, rather than accept it uncritically. Depending on the perturbation, it should reject unreliable evidence, correct erroneous intermediate reasoning, or reconsider conclusions affected by unsupported claims. For U4 (Condition Omission), the agent is expected to recover the omitted condition from the preceding interaction or established scientific knowledge, rather than silently introduce an unsupported assumption. Thus, an alternative result obtained by accepting unreliable information or following invalid reasoning is an agent failure with respect to the original scientific task, not a different correct answer.

Third, the validity of y∗y^* is explicitly checked through our LLM-based and human validation process. The original scientific problem and its verified reference solution serve as fixed references throughout task and perturbation construction. Both validators examine whether each instance conforms to the intended perturbation type and modifies only the designated task-relevant elements. They further check scientific plausibility, consistency with the original problem and its cross-turn dependencies, and the absence of unintended scientific changes. In particular, they verify that an instance does not introduce a different scientific question, alter the intended outcome, or invalidate y∗y^*. Only instances that pass both validation stages are retained. Validation is completed before agent execution and is independent of the evaluated agent.

In the revised paper, we have clarified the objective-preserving design in Section 4.2 (Pages 5–6) and detailed the applicability, instantiation, and validation criteria in Appendix C (Pages 21–22).

AuthorsOpenReview ↗

Response to Reviewer BnoS

Weakness 2: Handling Missing or Conflicting Information

Response: We appreciate the reviewer’s comment. We clarify how scientific agents should handle missing or conflicting information and introduce a per-turn self-reflection mechanism to investigate whether such issues can be detected and mitigated during multi-turn problem solving.

Reliable agents should distinguish recoverable information deficiencies from unresolved uncertainties. Specifically, they should recover missing information when it can be inferred from preceding interactions or established scientific knowledge, verify and correct unreliable or conflicting evidence, and revise invalid reasoning. When the available information is insufficient for reliable recovery, they should explicitly recognize the limitation and request clarification rather than proceed with unsupported assumptions. To investigate this capability, we introduce a per-turn self-reflection and repair mechanism. After each interaction turn, the agent reviews its latest response and accumulated interaction history to identify missing conditions, conflicting evidence, and potential reasoning errors. It then attempts to recover missing information, verify and correct unreliable evidence, or revise invalid reasoning. When reliable correction is not possible, it is instructed to request clarification. Importantly, the reflection mechanism has no access to perturbation types, injection positions, or reference solutions.

We evaluate this mechanism on 200 randomly sampled SciARP questions spanning all 13 perturbation types, using DeepSeek-V4-Flash and GPT-5.6. Under matched experimental settings, we compare the original agents with their reflection-enhanced counterparts while keeping the models, tasks, perturbations, and tools identical. As shown in the table below, self-reflection improves all four metrics, with average gains of 13.64, 7.10, 5.80, and 9.68 percentage points in TACC, IC, TP, and RV, respectively. The largest improvement is observed in TACC, followed by RV, indicating that reflection helps improve both end-to-end task success and intermediate reasoning validity. However, its effectiveness varies across models and remains insufficient to fully restore robustness.

ModelTACCICTPRV
DeepSeek-V4-Flash54.15 (+11.20)77.76 (+7.51)79.44 (+5.68)70.78 (+10.77)
GPT-5.660.56 (+16.07)85.43 (+6.69)86.32 (+5.91)74.27 (+8.59)
Average57.36 (+13.64)81.60 (+7.10)82.88 (+5.80)72.53 (+9.68)

In the revised paper, we have added the self-reflection and repair mechanism and its experimental evaluation in Section 5.5 (Page 10) and reported the corresponding results in Table 3 (Page 10).

AuthorsOpenReview ↗

Response to Reviewer BnoS

Weakness 3: Perturbation Strength and Category Comparability

Response: We thank the reviewer for this comment. The 13 perturbation types target qualitatively different aspects of scientific problem solving. Their impacts reflect distinct failure mechanisms rather than different levels on a unified perturbation scale. We therefore examine their effects both across categories and at different strength levels within each perturbation type.

First, the impact of perturbations is inherently type-dependent, as different perturbation mechanisms affect distinct capabilities and lead to different degrees of performance degradation. As shown in Table 2, reasoning perturbations cause the largest degradation in TACC (50.72 percentage points) and TP (22.76 percentage points), whereas conclusion perturbations most strongly affect IC (22.32 percentage points) and RV (31.66 percentage points). Evidence perturbations, in contrast, substantially reduce TACC (21.77 percentage points) while causing only a small reduction in TP (1.19 percentage points). These results demonstrate that different perturbation mechanisms affect distinct aspects of scientific problem solving.

Second, we systematically investigate perturbation strength using three mechanism-specific levels: weak, moderate, and strong. Rather than applying a uniform modification rule across different perturbation types, we define three strength levels independently for each type according to its underlying mechanism. These levels progressively increase the extent of the corresponding perturbation while preserving its fundamental mechanism and the original scientific objective. Thus, strength is compared within each perturbation type, without assuming that the same level represents equivalent absolute severity across different types. For example, for U4 (Condition Omission), the weak level omits the reagent stoichiometry while retaining the instruction to react with both carbonyl groups, whereas the strong level additionally removes the explicit specification of the reaction scope, leaving it unclear whether one or both carbonyl groups should react. We evaluate all three levels using TACC, IC, TP, and RV.

The strength analysis reveals consistently increasing degradation with perturbation strength. For each of the 13 perturbation types, the eight-model average degradation increases monotonically from weak to moderate and strong levels across all four metrics. For example, average TACC degradation increases from 18.41 to 66.86 and 92.73 percentage points for E3, while U4 exhibits a pronounced strength-dependent increase from 2.25 to 15.70 and 75.25 percentage points. These results further demonstrate that agents exhibit heterogeneous sensitivity to different perturbation mechanisms, with stronger perturbations generally causing greater degradation.

In the revised paper, we have added the three-level perturbation strength analysis to Section 5.4 (Page 9) and provided the detailed results for all 13 perturbation types in Appendix J (Page 38, Tables 28–31, Pages 40–41).

Question 1: Statistical Significance Analysis

Response: We appreciate the reviewer’s helpful question. We have conducted additional paired statistical analyses to assess the significance and reliability of perturbation-induced performance degradation.

Specifically, for each task and model, we average the scores across three independent runs under each condition and calculate the paired difference between clean and perturbed executions. We estimate 95% confidence intervals using 10,000 paired bootstrap resamples over task IDs and perform two-sided paired permutation tests using 1,000,000 random sign-flip permutations. Holm–Bonferroni correction is applied separately for each evaluation metric across the 13 perturbation types. The results show that all 52 comparisons (13 perturbation types × 4 metrics) have positive pointwise 95% bootstrap confidence intervals excluding zero and remain statistically significant after Holm–Bonferroni correction, with adjusted p-values ranging from 0.000013 to 0.023432. In the revised paper, we have added the statistical analysis protocol and corresponding results in Appendix F (Page 36).

Reviewer_ej8HOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper constructs a benchmark for testing whether LLM-based scientific agents remain reliable when plausible errors or misleading information arise during multi-turn scientific problem solving. It converts 620 science problems into 3–13-turn tasks and injects 13 perturbation types across four stages: problem understanding, evidence processing, reasoning, and conclusion formation. It compares clean and perturbed live agent trajectories using final task success plus process metrics for information correctness, task progression, and reasoning validity. Across eight models, the paper finds that perturbations substantially degrade performance, especially reasoning-related ones. It also shows that agents may continue progressing through a task despite using incorrect information, stronger clean-task performance does not necessarily mean better robustness, and errors can emerge later, propagate across turns, and rarely self-correct.

Critical strengths
  1. This paper studies an underexplored yet important problem: whether scientific agents can remain reliable when plausible perturbations arise at different stages of multi-turn scientific problem solving. This setting is highly relevant because errors introduced early in a workflow can affect later reasoning and final conclusions.
  2. The proposed perturbation taxonomy is well structured. It covers multiple stages of scientific problem solving and applies perturbations within multi-turn interactions, rather than treating robustness solely as sensitivity to a modified initial prompt.
  3. The paired evaluation of clean and perturbed executions is a strong design choice. Comparing matched trajectories can reveal not only changes in final task performance, but also when perturbation-induced failures emerge and whether they persist or propagate.
Critical weaknesses
  1. The authors should analyze contamination and memorization risks, since GPQA, OlympiadBench, and other public benchmarks may have appeared in model training or post-training data. This risk applies to both clean performance and robustness estimates.
  2. The author should show the variance/standard deviations, confidence intervals, p-values, paired significance tests etc if the results are from multiple runs.
Questions for authors

How is the perturbation guaranteed to preserve the original task objective rather than subtly changing the problem being solved?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ej8H

We sincerely thank the reviewer for the positive assessment of SciARP, particularly for recognizing the importance of the research problem, the well-structured perturbation taxonomy, and the paired evaluation design. We also appreciate the constructive feedback, which has helped us further strengthen the benchmark. In response, we have expanded SciARP from 620 to 720 tasks by incorporating 100 additional physics problems from PhysLogic, conducted three independent runs with paired statistical significance analyses, and further clarified the preservation of the original scientific objective through fixed reference solutions and LLM–human dual validation. All modifications in the revised manuscript are highlighted in blue.

Weakness 1: Potential Contamination and Memorization Risks

Response: We appreciate the reviewer’s helpful comment. To address potential contamination and memorization risks, we have expanded the benchmark with 100 additional physics problems, constructed new multi-turn tasks from the source problems, and clarified how our paired clean–perturbed evaluation controls for task-specific familiarity.

First, SciARP does not directly evaluate the original benchmark questions in their single-turn form. Instead, we transform each source problem into a newly constructed multi-turn task. An LLM-based Task Planner decomposes its verified solution into 3–13 logically connected subtasks, introducing newly constructed intermediate reasoning steps and explicit cross-turn dependencies. Thus, successful completion requires agents to maintain and correctly reuse scientific information throughout the interaction, rather than relying solely on memorized answers to the original questions.

Second, we expand SciARP with 100 additional physics reasoning problems from PhysLogic (Yu et al., 2026), a recently published ICML 2026 benchmark. Compared with the four established and widely used benchmarks, this recent source makes prior exposure to these problems less likely, although it cannot be entirely ruled out. The selected source problems have verified reference solutions, independently verifiable correctness, and require non-trivial multi-step reasoning. The resulting multi-turn tasks undergo the same LLM-based and human dual-validation procedure, following predefined quality criteria.

Third, our paired clean–perturbed evaluation helps control for task-specific familiarity when measuring perturbation-induced degradation. Each clean–perturbed pair is derived from the same underlying scientific problem, shares the same multi-turn task structure and verified reference solution, and is evaluated using the same model under matched experimental settings. Consequently, any prior familiarity with the underlying problem is shared across the paired conditions, helping control for task-specific familiarity when estimating the performance difference attributable to perturbations.

After incorporating 100 additional physics problems from the recently published PhysLogic benchmark, we recompute the experiments on the expanded 720-task benchmark. Averaged equally across the 13 perturbation types and eight evaluated models, TACC decreases from 90.90% under clean conditions to 64.74% under perturbed conditions, corresponding to a degradation of 26.16 percentage points. We observe consistent degradation in IC (97.14% to 88.26%), TP (97.52% to 88.56%), and RV (95.27% to 80.16%). Importantly, the expanded benchmark preserves our key finding that different perturbation stages exhibit distinct robustness profiles: reasoning perturbations cause the largest degradation in TACC and TP (50.72 and 22.76 percentage points, respectively), whereas conclusion perturbations most strongly affect IC and RV (22.32 and 31.66 percentage points, respectively).

In the revised paper, we have revised the benchmark construction description in Section 1 (Page 2), Section 4.2 (Pages 5–6), and Appendix A (Pages 16–17) to clarify the five data sources, multi-turn task construction, and validation procedure. Figure 1 (Page 5) has also been updated to reflect the expanded benchmark of 720 tasks. The corresponding experimental results have been updated in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer ej8H

Weakness 2: Statistical Reliability and Significance of Experimental Results

Response: We appreciate the reviewer’s helpful comment. We have extended our statistical analysis to quantify execution variability and assess the significance of perturbation-induced degradation.

First, we quantify execution variability across independent runs. Each clean and perturbed task is independently executed three times under identical experimental settings, with each repetition constituting a fresh live run. We report the mean performance across runs and calculate standard deviations to quantify execution variability. The SD ranges across models and perturbation types are summarized in Appendix F of the revised paper.

Second, we conduct paired statistical analyses to assess the significance of perturbation-induced degradation. For each task and model, we average scores across three runs under each condition and calculate the paired difference between clean and perturbed executions. We aggregate task-level differences across the eight models using equal weights. We estimate 95% confidence intervals using 10,000 paired bootstrap resamples over task IDs and perform two-sided paired permutation tests using 1,000,000 random sign-flip permutations. Holm–Bonferroni correction is applied separately for each evaluation metric across the 13 perturbation types. The results show consistent and statistically significant degradation across all 52 comparisons. All pointwise 95% bootstrap confidence intervals exclude zero, and all comparisons remain significant after Holm–Bonferroni correction, with adjusted p-values ranging from 0.000013 to 0.023432. The standard deviations across three runs range from 0.11 to 2.36 percentage points for TACC, 0.12 to 0.69 for IC, 0.06 to 0.37 for TP, and 0.22 to 1.72 for RV.

In the revised paper, we have clarified the repeated-run protocol in Section 5.1 (Page 7) and added the statistical analysis protocol and corresponding results in Appendix F (Page 36).

AuthorsOpenReview ↗

Response to Reviewer ej8H

Question: Preservation of the Original Scientific Objective

Response: We thank the reviewer for this comment. Preserving the original scientific objective and ground truth is a fundamental constraint in SciARP. We enforce this constraint through fixed task references, stage-specific perturbation rules, and LLM–human dual validation.

First, the original scientific problem and its verified reference solution y∗y^* serve as fixed references throughout perturbation construction and evaluation. Neither is modified by any of the 13 perturbation types. Perturbations are introduced only into designated intermediate information or reasoning elements. The original problem and final task objective remain unchanged between the clean and perturbed versions. During evaluation, perturbed information is treated as context to be assessed, rather than as an authoritative replacement for the original ground truth. Thus, SciARP therefore measures whether agents solve the same task reliably under imperfect information, not whether they solve a modified task.

Second, each perturbation is constructed under stage-specific constraints to modify the problem-solving process without changing the underlying task. The Perturbation Generator modifies only designated elements while preserving the original problem, scientific objective, and verified reference solution. Specifically:

Problem understanding (U1–U4): Perturbations modify the representation or availability of task-relevant information, including terminology, units, variable notations, and task conditions, without redefining the original scientific problem or its intended conditions.

Evidence processing (E1–E4): Perturbations introduce unreliable premises, citation or attribution errors, data or statistical inconsistencies, or conflicting information from multiple sources. They modify only the designated intermediate evidence or its presentation, while retaining the original scientific problem and reference evidence as the basis for evaluating information validity.

Reasoning (R1–R3): Perturbations introduce cross-step inconsistencies, unsupported causal inferences, or misleading reasoning paths into intermediate problem solving. They affect the reasoning process rather than the original problem conditions or scientifically valid solution, allowing the introduced deviations to be assessed against the original task and its reference solution.

Conclusion formation (C1–C2): Perturbations introduce pressure to suppress uncertainty or accept unsupported authority and stance signals. They modify the contextual cues influencing scientific judgment without changing the underlying scientific evidence or its implications.

Across all four stages, the original scientific problem and verified reference solution remain fixed. The perturbations modify only designated information, reasoning elements, or contextual cues encountered by the agent, rather than introducing a different scientific objective or ground truth.

Third, every perturbed instance undergoes both LLM-based and human validation. The validators compare each instance against the original problem and y∗y^*. They check perturbation correctness, scientific plausibility, scope of modification, dependency validity, and the absence of unintended scientific changes. Any instance that changes the scientific question, alters the intended outcome, or establishes a different ground truth is rejected. Only instances that pass both stages are retained. Validation is completed before any agent execution and is independent of the evaluated model.

In the revised paper, we have clarified the objective-preserving perturbation construction in Section 4.2 (Pages 5–6) and provided detailed applicability, instantiation, and dual-validation criteria in Appendix C (Pages 21–22).

We would be happy to provide further clarification and address any additional questions the reviewer may have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Thank You for the Constructive Reviews

We thank all reviewers for their constructive and valuable feedback. Below, we summarize how we have addressed your main concerns in our responses and the revised manuscript (with all changes highlighted in blue in the revised submission):

  1. Benchmark scale and contamination (Reviewer ej8H, ruSN): We have expanded SciARP from 620 to 720 tasks by adding 100 physics problems from the recent PhysLogic benchmark and re-ran all experiments (Section 4.2, Appendix A). We also clarify how transforming single-turn problems into newly constructed 3–13-turn interdependent tasks and using a paired clean–perturbed evaluation design mitigate prior memorization and control for task-specific familiarity.

  2. Statistical reliability (Reviewer ej8H, BnoS): We now report results across three independent live runs with standard deviations (Section 5.1, Appendix F). Paired bootstrap and two-sided permutation tests (Holm–Bonferroni corrected) confirm statistically significant degradation across all 52 comparisons (13 perturbation types×4 metrics13 \text{ perturbation types} \times 4 \text{ metrics}).

  3. Ground-truth preservation (Reviewer ej8H, BnoS): We clarify that the original scientific problem and its verified reference solution (y∗y^*) remain fixed throughout construction and evaluation (Section 4.2, Appendix C). Every perturbation is constrained by stage-specific rules and undergoes LLM–human dual validation to ensure it does not alter the underlying scientific objective or ground truth.

  4. Self-reflection, perturbation strength, and failure attribution (Reviewer BnoS): We added: (1) a per-turn self-reflection and repair mechanism (Section 5.5, Table 3), which improves TACC by +13.64 percentage points on average across DeepSeek-V4-Flash and GPT-5.6 though without fully restoring robustness; (2) a three-level (weak, moderate, strong) perturbation strength analysis showing monotonic degradation increases across all 13 types (Section 5.4, Appendix J); (3) a five-category failure trajectory attribution analysis distinguishing information adoption errors from reasoning and execution failures (Appendix G, Table 19); and (4) a clarification of our unified evaluation and macro-averaging protocol across categories (Appendix D).

  5. Human validation and evaluation (Reviewer BnoS, ruSN): We detail our dual-validation and evaluation protocols involving two blinded expert annotators, each with at least five years of scientific research experience, working from predefined rubrics (Appendices A, C, and D). Disagreements are resolved via consensus adjudication during benchmark construction, and via strict human–LLM agreement during trajectory evaluation.

  6. Perturbation realism and motivation (Reviewer ruSN): We added literature-grounded motivations and real-world scientific examples for all 13 perturbation types in Appendix B, and highlighted how U2 and U3 target cross-step dimensional and symbolic consistency (degrading TACC by 5.52 and 5.58 percentage points, with effects persisting to the final turn in 62% and 66% of cases).

  7. Presentation (Reviewer ruSN): We revised Figure 1 with larger fonts, simplified text, and an updated layout reflecting the expanded 720-task benchmark.

As the discussion period draws to a close, we would like to take this opportunity to express our sincere gratitude to all reviewers for your time and thoughtful feedback. We would be very pleased to provide further clarification and answer any additional questions.

Authors of Submission 20328

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.