Weaknesses
1. Substantial problems with clarity limit assessment of the contributions
I found the paper exceptionally difficult to follow due to several presentation issues, including but not limited to: convoluted language; dense combinations of undefined terms and jargon; explanations that too often defer details to the appendices; unnecessarily complicated language to explain simple concepts; unclear interpretation of findings; explanations of historical and/or failed experiments with little to no context.
In my assessment, these issues are substantial enough to warrant rejection, as they prevent a reliable evaluation of the methods, results, contributions, and the paper's overall significance.
I provide examples at the end. I strongly encourage the authors to revise the manuscript extensively, especially if substantive parts of the text was written with the assistance of AI.
2. Unclear practical relevance and significance
I have substantial concerns about the relevance of the identified issues and the proposed solutions, particularly given that some have already been addressed in existing benchmark implementations.
For example, the authors explicitly acknowledge that the successor to τ-bench, τ²-bench, already avoids the identified problem by performing evaluation in a separate environment:
In an exploratory replay, forked calls change nothing (Appendix C.2). The successor, τ 2-bench (Barres et al., 2025), grades in a freshly built environment and closes this channel.
Similarly, the WebArena issue has been identified and resolved in subsequent works and/or git fixes of the original code [1], [2], [3]
These observations raise questions about the practical significance and novelty of the contributions. The paper would benefit from more convincingly establishing whether the identified problems remain relevant to current agent evaluation and training pipelines and, importantly, whether these issues are not already well understood by practitioners in the field.
3. Limited empirical evidence supporting broader claims
As the authors themselves acknowledge in §7, their experiments have several substantial limitations (quoted below, with emphasis added; the passage also illustrates my concerns regarding clarity):
Our AppWorld live runs are too small to estimate how often this happens with live agents (repeat floors 2 and 0 of 55). The closed-loop τ-bench contrasts give size and direction only (§3.1). The published AppWorld runs graded only at the end, and no published result we found used the analysis script, so we show no published number to be wrong. In both live WebArena runs the agent solved too few tasks to pass our competence gate, so the browser claim stops at disturbance. Six pre-declared gates, two predictions fixed in advance and part of a third failed, each reported beside the claim it limits (Table 18).
Given these limitations, could the authors clarify:
- Practical significance: The experiments appear to provide limited evidence regarding the prevalence of evaluator interference or its practical consequences for contemporary agent research. How do the authors justify the broader significance of their findings?
- Experimental limitations: What exactly do the authors mean by "six pre-declared gates" and the final sentence of the quoted passage? Could they explain more clearly which experimental criteria and predictions were not satisfied and how these failures affect the conclusions of the paper?
4. Limited generalizability of the causal analysis
The causal investigation in AppWorld primarily identifies specific shared-state modifications responsible for the observed evaluation errors and verifies that restoring those variables eliminates the errors.
Although these experiments help establish the causes of the observed failures, it is unclear what generalizable methodological insights they provide beyond debugging the particular benchmark implementation.
The paper would benefit from clarifying how this analysis advances our understanding of evaluator interference beyond identifying and correcting implementation-specific bugs.
5. Limited theoretical and methodological novelty
The formalization of evaluator non-interference is relatively straightforward, essentially requiring that additional evaluation calls leave recorded outcomes unchanged.
Similarly, Theorem 1 restates established operating-system properties of process isolation, as the authors explicitly acknowledge:
The theorem restates ordinary process semantics; what we add is a check of its condition on the running system (Appendix K).
The proposed mitigation, running evaluators in isolated processes, is also a well-established software-engineering practice.
Could the authors clarify what their work contributes beyond applying existing isolation techniques to specific benchmarks? Could the authors demonstrate that the proposed formalization and analysis provide a generalizable approach to identifying evaluator interference that goes beyond established software-testing practices and is applicable more generally across benchmarks?
Examples of clarity issues
The following are some excerpts that I found difficult to follow. They represent only a subset of the issues; I strongly encourage a complete revision of the manuscript.
Introduction
Finally we ask what protects a pipeline (§5). Running the grader in a disposable copy of its process, a fork, keeps its writes out of the agent’s memory except through files, under a condition we check rather than assume. Against an ungraded replay it changes no outcome beyond the floor, at a cost we report. Reusing the first grade repairs the τ-bench pipeline, and we say which leaks neither fix closes.
This passage combines several technical concepts and conclusions without first explaining them concretely. Expressions such as "beyond the floor," "under a condition we check rather than assume," and "which leaks neither fix closes" make the summary unnecessarily difficult to interpret.
Section 3.1
All 105 excess passes are tasks the benchmark failed on its database check, and no pass becomes a failure. Our protocol predicted that every excess pass would fall on a task without an output check, the part of the reward that looks for expected strings in the agent’s replies. Of the 105, 100 do and 5 do not, so that prediction failed. The 5 are retail tasks whose expected outputs the agent had stated. A diagnostic made afterwards, with no model call, says why. Whether the agent’s messages contain the expected outputs reproduces the pipeline’s verdict on all 159 episodes. After the first call, the second no longer tests whether the agent changed the database.
Could the authors clarify this overall passage? Which protocol is being referenced, what motivates the prediction that every excess pass would occur on a task without an output check, and why is this prediction relevant? What are the implications of the failed prediction for the paper's findings? Does it undermine any of the paper conclusions?
This is the paper's main results section (3.1):
The smallest repair is to reuse the first grade, which agrees with the benchmark by construction. We ran it in the same pipeline on new episodes to check the plumbing. All 161 episodes that ended returned their cached grade, three step-capped episodes were graded once on a copy, and one episode stopped on a model-service error. The rubric’s call never changed the original data.
Could the authors clarify:
- What does it mean to "check the plumbing"?
- What is the significance of "and one episode stopped on a model-service error" to the interpretation of this experiment?
- Could they explain "The rubric’s call never changed the original data."?
Grading in process after every step adds between 29 and 79 passes across four configurations, from 42 to 117 for Qwen3-14B
The presentation does not immediately make clear what is being compared, what the reported differences represent, or how these findings should be interpreted given the variability acknowledged later in the paragraph.
Section 3.2
AppWorld’s (Trivedi et al., 2024a) public evaluate method, served over HTTP beside the action endpoint, grades a live world from inside it, whereas its offline batch evaluator refuses to start while a world is open.
What the authors mean by "whereas its offline batch evaluator refuses to start while a world is open."? What does it mean to "grad a live world from inside it"?
Of the 672 published evaluations, 666 are rebuilt, 662 reproduce the published evaluation, and 526 also pass the census’s eligibility. Three of the four runs fall below the 80% eligibility our protocol required. On those 526, grading after every action changes 36 evaluations, destroys 17 of the 142 published successes and manufactures none. Because eligibility already requires the forked arm to agree, the independent control uses all 666 rebuilt tasks. There in-process grading changes 45 evaluations, while the forked arm and a repeated ungraded replay each differ from the ungraded arm on 2, and none of those 4 changes a success. The published logs keep each API request but not the code that issued it, so we grade after every request, a finer schedule than the agent’s own steps
What does it mean to "pass the census's eligibility"? What does it mean to "fall below the 80% eligibility our protocol required."? What are the "published logs"? Could the authors clarify or write in more direct language this overall passage?
Sections 4.1 and 4.2
Could the authors clarify the claim and conclusion in the following passages?
Its two arms differ only in whether the per-step grading call is isolated, and a second run of the isolated arm is the floor, which disagrees with the first on 2 of 55 families. The agent passes 22 of 55 families isolated and 20 in process. Its graded arm differs on all 7 families the census flags and on 1 of the others, which we call cleared. Two of these eight also changed between the two isolated runs, so they are within the floor (Table 3). Of the seven, three lose a success and none gains one, as under HTTP, and one cleared family gains a success within the floor. These are counts on fixed families, and a family that matched itself once may still vary. All three score with the offline evaluator.
A second model family keeps the direction but not the clean split. With Llama-3.3-70B-Instruct in place of Qwen3-32B, the census flags 4 of the 55 families. The graded arm moves all 4 and 2 of the 51 cleared ones against a floor of 0, destroying 2 successes and manufacturing none. Its protocol, fixed before the run, predicted that no cleared family would move, so that part failed (Appendix B). Replaying the live agent’s own actions explains both moves, and both families are flagged on the Qwen trajectories. In one, the second channel of §4.3 acts on actions the census never replayed. In the other, grading changes what one of the agent’s actions returns, and the agent then acts differently.
In this sample the row decides whether a change is possible, and the placement decides only whether it shows. Grading once is therefore not safe. What is safe is grading when nothing that reads the written state remains (§4.3). Our data locate that point only roughly, within the last two actions.
Section 5
Theorem 1 is presented with limited explanation of why a formal theorem is necessary for the proposed solution, particularly given that the authors acknowledge it restates established operating-system semantics. Although a proof is provided in Appendix K, the main text would benefit from a clearer discussion of the theorem's purpose, significance, and relationship to the experimental contributions.
Other repairs. A cached first grade repairs the τ-bench report but not the first call’s write, and AppWorld’s snapshot endpoints isolate the grader but leave the world unclosable (Appendix H).
Could the authors explain more concretely what each repair addresses, why the alternatives are insufficient, and what the experiments establish about the proposed isolation approach? If these are significant, could they move material from the appendix to the main text?
References
[1] BrowserGym https://github.com/servicenow/browsergym
[2] The release of (Visual)WebArena in Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification (ICLR26)
[3] This less 10 lines fix to WebArena original code https://github.com/gasse/webarena/commit/a641a0b5b144d10e54471f5c827fa1484f76df5d
Response to Reviewer MtRM
Thank you for the review, and for the summary of each section. The paper took more effort to read than it should have, and it has been rewritten (see the general response). Short answer
In detail
We would be grateful if you would reconsider your rating in light of the rewritten paper.