acceptodds
Under review as a conference paper at ICLR 2027

The Grader Is an Intervention: Evaluator Non-Interference for Stateful Agents

Abstract

Benchmarks for AI agents score an agent by checking the environment it leaves behind, assuming the check changes nothing. Training, search and early stopping now grade episodes in progress or more than once. We show that a grader can score the current state correctly and still change a later score. We call the requirement that extra grading leave recorded outcomes unchanged evaluator non-interference, and test it in shipped code. Computing τ-bench’s reward writes the correct answer into the environment, so a public training wrapper that grades twice reports 127 passes where the benchmark reports 22. On AppWorld, grading after every API call turns 17 of 142 published successes into failures, and the benchmark’s own analysis script destroys 8. There, one grading call overwrites shared state in two ways, and undoing each repairs exactly its own cases, even on held-out tasks. With WebArena’s original evaluator, though not its PyPI port, Browser-Gym’s adapter lets the grader move the agent’s page. In a disposable copy of the environment’s process, grading changes no replayed outcome beyond run-to-run noise. Grading is therefore part of the experiment, and benchmarks should state when their grader runs and what reads the state it writes.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, stateful environments, benchmark validity, evaluator non-interference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MtRMOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how the existing graders in the benchmark could fail by interacting with the target system under the measurement.

1 Introduction

  • The paper illustrates the failure modes in the current grader on 3 benchmarks.
  • Line (91-99) basically claims that the current grader destroys the target systems by corrupting the records in two ways (Line 95). It also mentions how this could be fixed, running the grader in a disposable copy (Line 97).

3 Interference In Shipped Pipelines

  • Here, contract means that the grader doesn't change the target measurement.
  • '\tau-bench', AppWorld, and BrowserGym are examined, showing the issues in the existing graders.

4 AppWorld and 5 Repairs

  • This section shows the diagnosis in the failure cases and identifies the cause of the failure. The treatment to fix follows.
Critical strengths

I think the value of this paper is that it provides verification on the graders in many of the popular benchmarks in the literature. Seeing that there are notable issues in the grading systems and proposals for fixing them is a valuable contribution. However, I don't work on the benchmarks listed in the paper, so my assessment could be limited. I would read this paper carefully when working on those benchmarks.

Critical weaknesses

The main weakness of this paper is that the contribution of the work is more like fixing the issues in the grading systems in the benchmarks. This is obviously a good contribution in general, but I am not certain that this type of contribution is counted toward technical contributions for the main research track.

Questions for authors

I don't have questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MtRM

Thank you for the review, and for the summary of each section. The paper took more effort to read than it should have, and it has been rewritten (see the general response). Short answer

  • Response to the weakness (is this a main-track contribution?). We see the paper as a study of measurement validity, not a set of patches. It tests an assumption that stateful-agent benchmarks rely on, that grading only reads the environment, and shows that it fails in released code.

In detail

  • A general requirement. Section 2 states evaluator non-interference and the three conditions under which it can fail. They apply to any benchmark whose grader runs on a live environment. Table 1 applies them to seventeen environments.
  • A general test. Run the same episodes with and without the extra grading calls, and repeat one run to measure noise. It needs no knowledge of the grader's internals.
  • Evidence beyond one bug. Three benchmarks are affected for three different reasons: a database (τ-bench), process state (AppWorld) and a browser tab (WebArena). The effects are large: 127 passes reported instead of 22, and 17 of 142 published successes lost.
  • Why now. Training with step-level rewards and search call graders during an episode. The errors we observed run in one direction, which biases a reward, and averaging does not remove such a bias.

We would be grateful if you would reconsider your rating in light of the rewritten paper.

Thanks very much for your response.

I agree with the motivation of this study (inspect the grading of existing benchmarks and identify the issues) since post-training heavily rely on the graders. My original concern is this paper is in the shape of checking 3 benchmarks to identify sort of bugs in the grader, which is valuable but unclear it is significant to publish. So, my expectation was to get some diagnosis or standardized statements that can be applicable beyond three benchmarks. The response points to the three conditions in Section 2.2 and Section 2.3 as general requirements. However, it reads to me as a generic statement of reasoning. The observations in the paper are very task-specific.

Reviewer_fGfFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies evaluator non-interference for stateful-agent benchmarks: calling a grader should not change the episode’s recorded outcome or the state used by the agent. It formalizes this requirement and audits shipped implementations of τ-bench, AppWorld, and WebArena, BrowserGym, finding that graders can write shared state, create false passes or failures, and disturb the agent’s browser page. Controlled in-process, forked, repeated, and live-agent experiments show that isolated grader processes or environment copies can prevent these failures.

Critical strengths
  1. The paper identifies a fundamental and underexamined problem in stateful-agent evaluation: a grader may modify the state it is supposed to measure. Formalizing evaluator non-interference provides a clear correctness criterion for benchmarks that use repeated or online grading.

  2. The empirical audit is concrete and actionable. By testing shipped implementations of τ-bench, AppWorld, and WebArena/BrowserGym under ungraded, in-process, repeated, and isolated conditions, the paper separates grader-induced changes from normal run-to-run noise and demonstrates practical fixes.

Critical weaknesses
  1. The central requirement is intuitive and primarily an engineering invariant for benchmark and RL pipeline design. The paper does not establish how widespread the problem is under standard end-of-episode evaluation; indeed, the authors acknowledge that they do not show any published AppWorld result to be incorrect. Thus, the broader impact of the reported bugs remains unclear.

  2. Most standard benchmark protocols evaluate an agent only after the episode ends and then reset to a clean environment. The paper’s strongest failures arise in pipelines that perform intermediate or repeated grading, where non-interference is indeed a design requirement. However, the paper does not quantify how common these workflows are or show that standard leaderboard evaluations are affected, which limits the generality of its conclusions.

Questions for authors
  1. Under a strict final-only protocol with a fresh process and environment for each task, do any of the reported interference failures remain?

  2. How common are intermediate or repeated grading workflows in real training and evaluation pipelines, and do any standard leaderboard results change when rerun with fully isolated grading?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fGfF

Thank you for the review, and for stating the paper's limits so precisely. We agree with them, and the revised introduction now states them (see the general response). Numbers refer to the revised PDF.

Short answers

  • Response to the weaknesses. Agreed on both. The requirement is simple, and standard end-of-episode evaluation is safe. We do not show a published number to be wrong.
  • Answer to Question 1. With exactly one grading call after the last action and a fresh environment per task, none of the failures remained in our replays. One failure needs no mid-episode grading, however: a public τ-bench pipeline grades only at the end, but twice.
  • Answer to Question 2. We did not measure how common these workflows are. Re-grading AppWorld's published runs in isolation changes no published success.

Answer to Question 1, in detail

  • One call at the end. It changed 0 of 165 outcomes. A single call followed by at most two actions changed 0 of 208 replays (Section 4.3).
  • Two calls at the end. The public τ-bench training environment computes the reward again after the episode. It reports 127 passes on 159 episodes where the benchmark's own grade gives 22 (Section 3.1).
  • Fresh environment per task. On AppWorld the damage does not carry over: a world built after a graded one was closed was unaffected in 55 of 55 tasks (Section 5).

Answer to Question 2, in detail

  • Do such workflows exist? Yes, in released code: AppWorld's documentation suggests calling the grader "at any point" for RL rewards; AppWorld's own analysis script grades after every five interactions; BrowserGym's adapter calls WebArena's evaluator after every step; the τ-bench environment above grades twice.
  • How common? Not measured, and Section 7 says so. Of seventeen environments we read, five have a grader that acts on the live environment (new Table 1).
  • Leaderboards. AppWorld's published runs graded once, at the end. We rebuilt 666 of the 672 published evaluations. Grading in a separate process changed 2 of them, as many as simply repeating the replay, and no success was affected. On the 526 reproducible tasks, the same runs lose 17 of 142 successes if the grader is also called after every API request (Section 3.2).

Response to the weaknesses, in detail

  • An engineering invariant. Yes. Our point is that the released graders of three benchmarks break it, with no error message. Step-level rewards and search are where such graders are now being called.
  • What a user gains. The three conditions of Section 2.2 tell a pipeline author when grading is safe. The test costs two runs.

We would be grateful if you would reconsider your scores in light of the rewritten paper.

Thanks for the clarification. I’ve raised my soundness score from 2 to 3. I still have some concerns about the overall significance and scope of the contribution, so I’ve kept the other scores unchanged.

Reviewer_34AwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

This paper investigates whether evaluators ("graders") in existing agent benchmarks can inadvertently affect performance metrics or agent behavior. The authors distinguish between evaluator non-interference, which requires additional grading calls not to change recorded outcomes, and non-disturbance, which requires grading not to alter the environment state the agent interacts with.

The authors conduct experiments comparing executions with and without intermediate call to evaluator scripts, as well as evaluators running on isolated processes. They identify issues in three benchmarks: 1) in τ-bench, repeated grading inflate reported scores as the evaluator modifies the environment by replaying the ground-truth solution. 2) In WebArena, the original evaluator can navigate the agent's active browser page, altering the environment from which the agent continues executing. 3) In AppWorld, intermediate grading and the benchmark's analysis script modify shared bookkeeping state, causing agent work to be overlooked or lost and artificially reducing success rates. They further investigate the mechanisms of interference in AppWorld by selectively restoring state modified by the evaluator.

Finally, they propose running graders in isolated, forked processes as a mitigation and experimentally demonstrate that this can eliminate grader interference.

Critical strengths
  • The paper identifies concrete evaluation issues in relevant agent benchmarks, illustrating how grader side effects can distort reported performance.
  • The paper cleanly separate two ways in which evaluators can affect agent evaluation: directly, by modifying the state used to compute performance metrics, and indirectly, by altering the environment state from which the agent subsequently acts.
Critical weaknesses

Weaknesses

1. Substantial problems with clarity limit assessment of the contributions

I found the paper exceptionally difficult to follow due to several presentation issues, including but not limited to: convoluted language; dense combinations of undefined terms and jargon; explanations that too often defer details to the appendices; unnecessarily complicated language to explain simple concepts; unclear interpretation of findings; explanations of historical and/or failed experiments with little to no context.

In my assessment, these issues are substantial enough to warrant rejection, as they prevent a reliable evaluation of the methods, results, contributions, and the paper's overall significance.

I provide examples at the end. I strongly encourage the authors to revise the manuscript extensively, especially if substantive parts of the text was written with the assistance of AI.

2. Unclear practical relevance and significance

I have substantial concerns about the relevance of the identified issues and the proposed solutions, particularly given that some have already been addressed in existing benchmark implementations.

For example, the authors explicitly acknowledge that the successor to τ-bench, τ²-bench, already avoids the identified problem by performing evaluation in a separate environment:

In an exploratory replay, forked calls change nothing (Appendix C.2). The successor, τ 2-bench (Barres et al., 2025), grades in a freshly built environment and closes this channel.

Similarly, the WebArena issue has been identified and resolved in subsequent works and/or git fixes of the original code [1], [2], [3]

These observations raise questions about the practical significance and novelty of the contributions. The paper would benefit from more convincingly establishing whether the identified problems remain relevant to current agent evaluation and training pipelines and, importantly, whether these issues are not already well understood by practitioners in the field.

3. Limited empirical evidence supporting broader claims

As the authors themselves acknowledge in §7, their experiments have several substantial limitations (quoted below, with emphasis added; the passage also illustrates my concerns regarding clarity):

Our AppWorld live runs are too small to estimate how often this happens with live agents (repeat floors 2 and 0 of 55). The closed-loop τ-bench contrasts give size and direction only (§3.1). The published AppWorld runs graded only at the end, and no published result we found used the analysis script, so we show no published number to be wrong. In both live WebArena runs the agent solved too few tasks to pass our competence gate, so the browser claim stops at disturbance. Six pre-declared gates, two predictions fixed in advance and part of a third failed, each reported beside the claim it limits (Table 18).

Given these limitations, could the authors clarify:

  • Practical significance: The experiments appear to provide limited evidence regarding the prevalence of evaluator interference or its practical consequences for contemporary agent research. How do the authors justify the broader significance of their findings?
  • Experimental limitations: What exactly do the authors mean by "six pre-declared gates" and the final sentence of the quoted passage? Could they explain more clearly which experimental criteria and predictions were not satisfied and how these failures affect the conclusions of the paper?

4. Limited generalizability of the causal analysis

The causal investigation in AppWorld primarily identifies specific shared-state modifications responsible for the observed evaluation errors and verifies that restoring those variables eliminates the errors.

Although these experiments help establish the causes of the observed failures, it is unclear what generalizable methodological insights they provide beyond debugging the particular benchmark implementation.

The paper would benefit from clarifying how this analysis advances our understanding of evaluator interference beyond identifying and correcting implementation-specific bugs.

5. Limited theoretical and methodological novelty

The formalization of evaluator non-interference is relatively straightforward, essentially requiring that additional evaluation calls leave recorded outcomes unchanged.

Similarly, Theorem 1 restates established operating-system properties of process isolation, as the authors explicitly acknowledge:

The theorem restates ordinary process semantics; what we add is a check of its condition on the running system (Appendix K).

The proposed mitigation, running evaluators in isolated processes, is also a well-established software-engineering practice.

Could the authors clarify what their work contributes beyond applying existing isolation techniques to specific benchmarks? Could the authors demonstrate that the proposed formalization and analysis provide a generalizable approach to identifying evaluator interference that goes beyond established software-testing practices and is applicable more generally across benchmarks?

Examples of clarity issues

The following are some excerpts that I found difficult to follow. They represent only a subset of the issues; I strongly encourage a complete revision of the manuscript.

Introduction

Finally we ask what protects a pipeline (§5). Running the grader in a disposable copy of its process, a fork, keeps its writes out of the agent’s memory except through files, under a condition we check rather than assume. Against an ungraded replay it changes no outcome beyond the floor, at a cost we report. Reusing the first grade repairs the τ-bench pipeline, and we say which leaks neither fix closes.

This passage combines several technical concepts and conclusions without first explaining them concretely. Expressions such as "beyond the floor," "under a condition we check rather than assume," and "which leaks neither fix closes" make the summary unnecessarily difficult to interpret.

Section 3.1

All 105 excess passes are tasks the benchmark failed on its database check, and no pass becomes a failure. Our protocol predicted that every excess pass would fall on a task without an output check, the part of the reward that looks for expected strings in the agent’s replies. Of the 105, 100 do and 5 do not, so that prediction failed. The 5 are retail tasks whose expected outputs the agent had stated. A diagnostic made afterwards, with no model call, says why. Whether the agent’s messages contain the expected outputs reproduces the pipeline’s verdict on all 159 episodes. After the first call, the second no longer tests whether the agent changed the database.

Could the authors clarify this overall passage? Which protocol is being referenced, what motivates the prediction that every excess pass would occur on a task without an output check, and why is this prediction relevant? What are the implications of the failed prediction for the paper's findings? Does it undermine any of the paper conclusions?

This is the paper's main results section (3.1):

The smallest repair is to reuse the first grade, which agrees with the benchmark by construction. We ran it in the same pipeline on new episodes to check the plumbing. All 161 episodes that ended returned their cached grade, three step-capped episodes were graded once on a copy, and one episode stopped on a model-service error. The rubric’s call never changed the original data.

Could the authors clarify:

  • What does it mean to "check the plumbing"?
  • What is the significance of "and one episode stopped on a model-service error" to the interpretation of this experiment?
  • Could they explain "The rubric’s call never changed the original data."?

Grading in process after every step adds between 29 and 79 passes across four configurations, from 42 to 117 for Qwen3-14B

The presentation does not immediately make clear what is being compared, what the reported differences represent, or how these findings should be interpreted given the variability acknowledged later in the paragraph.

Section 3.2

AppWorld’s (Trivedi et al., 2024a) public evaluate method, served over HTTP beside the action endpoint, grades a live world from inside it, whereas its offline batch evaluator refuses to start while a world is open.

What the authors mean by "whereas its offline batch evaluator refuses to start while a world is open."? What does it mean to "grad a live world from inside it"?

Of the 672 published evaluations, 666 are rebuilt, 662 reproduce the published evaluation, and 526 also pass the census’s eligibility. Three of the four runs fall below the 80% eligibility our protocol required. On those 526, grading after every action changes 36 evaluations, destroys 17 of the 142 published successes and manufactures none. Because eligibility already requires the forked arm to agree, the independent control uses all 666 rebuilt tasks. There in-process grading changes 45 evaluations, while the forked arm and a repeated ungraded replay each differ from the ungraded arm on 2, and none of those 4 changes a success. The published logs keep each API request but not the code that issued it, so we grade after every request, a finer schedule than the agent’s own steps

What does it mean to "pass the census's eligibility"? What does it mean to "fall below the 80% eligibility our protocol required."? What are the "published logs"? Could the authors clarify or write in more direct language this overall passage?

Sections 4.1 and 4.2

Could the authors clarify the claim and conclusion in the following passages?

Its two arms differ only in whether the per-step grading call is isolated, and a second run of the isolated arm is the floor, which disagrees with the first on 2 of 55 families. The agent passes 22 of 55 families isolated and 20 in process. Its graded arm differs on all 7 families the census flags and on 1 of the others, which we call cleared. Two of these eight also changed between the two isolated runs, so they are within the floor (Table 3). Of the seven, three lose a success and none gains one, as under HTTP, and one cleared family gains a success within the floor. These are counts on fixed families, and a family that matched itself once may still vary. All three score with the offline evaluator.

A second model family keeps the direction but not the clean split. With Llama-3.3-70B-Instruct in place of Qwen3-32B, the census flags 4 of the 55 families. The graded arm moves all 4 and 2 of the 51 cleared ones against a floor of 0, destroying 2 successes and manufacturing none. Its protocol, fixed before the run, predicted that no cleared family would move, so that part failed (Appendix B). Replaying the live agent’s own actions explains both moves, and both families are flagged on the Qwen trajectories. In one, the second channel of §4.3 acts on actions the census never replayed. In the other, grading changes what one of the agent’s actions returns, and the agent then acts differently.

In this sample the row decides whether a change is possible, and the placement decides only whether it shows. Grading once is therefore not safe. What is safe is grading when nothing that reads the written state remains (§4.3). Our data locate that point only roughly, within the last two actions.

Section 5

Theorem 1 is presented with limited explanation of why a formal theorem is necessary for the proposed solution, particularly given that the authors acknowledge it restates established operating-system semantics. Although a proof is provided in Appendix K, the main text would benefit from a clearer discussion of the theorem's purpose, significance, and relationship to the experimental contributions.

Other repairs. A cached first grade repairs the τ-bench report but not the first call’s write, and AppWorld’s snapshot endpoints isolate the grader but leave the world unclosable (Appendix H).

Could the authors explain more concretely what each repair addresses, why the alternatives are insufficient, and what the experiments establish about the proposed isolation approach? If these are significant, could they move material from the appendix to the main text?

References

[1] BrowserGym https://github.com/servicenow/browsergym

[2] The release of (Visual)WebArena in Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification (ICLR26)

[3] This less 10 lines fix to WebArena original code https://github.com/gasse/webarena/commit/a641a0b5b144d10e54471f5c827fa1484f76df5d

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 34Aw

Thank you for the detailed review. Your list of passages was our checklist for the rewrite: each of them has been rewritten (see the general response). Numbers refer to the revised PDF.

Short answers

  • Response to Weakness 1 (clarity). You were right, and the main text is new.
  • Response to Weakness 2 (already fixed?). In part. τ²-bench and the evaluator that BrowserGym installs are not affected, and the paper says so. τ-bench's reward method, AppWorld's evaluate and WebArena's original evaluator are unchanged.
  • Response to Weakness 3 (evidence). We show that the problem exists and, on AppWorld, why. We do not show how common it is, and Section 7 says so.
  • Response to Weakness 4 (generality). The seven overwritten attributes are specific to AppWorld. The method and three observations are not (below).
  • Response to Weakness 5 (novelty). Agreed that the definition is simple and that isolation is standard. Theorem 1 has moved to the appendix. The contribution is empirical.

Response to Weakness 1, in detail: the passages you quoted

  • Failed prediction (Section 3.1). τ-bench's reward checks the database and, for some tasks, required strings in the agent's replies. We predicted that every extra pass would be on a task without required strings. 100 of the 105 are and 5 are not, so the prediction failed. The cause is the same for all 105: after the first call the database check compares the reference with itself. The conclusion (127 reported, 22 by the benchmark) does not depend on the prediction.
  • "Check the plumbing". We ran the fixed pipeline to check our implementation of the fix: 161 of 161 finished episodes returned the benchmark's grade.
  • "29 to 79 passes". Same 165 tasks, reward computed after every step, in the environment's process versus a separate process, four model configurations. Identical runs disagree on 4–36 tasks, so we claim direction and rough size only.
  • AppWorld (Section 3.2). AppWorld has an offline evaluator (used for the leaderboard) and an evaluate method that can run on a live world. "Reproducible" replaces "eligible" and is defined in Section 2.3. "Flagged", "cleared" and "floor" are gone.

Response to Weakness 2, in detail

  • WebArena. Thank you for the pointers. They agree with what we report: the fork that BrowserGym installs opens a disposable tab, so PyPI installs are not affected (Section 3.3, Table 1). The original repository still calls page.goto on the page it is given (main branch, checked again on 14 November 2026).
  • τ-bench. The successor is not affected. The original is what the public training environment we tested wraps.
  • AppWorld. Not fixed in the release we tested (0.1.3.post1). Its documentation suggests calling evaluate during an episode for RL rewards.
  • Well understood? By some: one τ-bench wrapper we cite scores a copy and says why. The cases above show that it is not understood everywhere.

Response to Weakness 3, in detail

  • "Gates". A gate was a threshold we wrote down before an experiment, such as "at least 10 successes". Six were missed, and each narrowed a claim: both live WebArena runs had 5 successes, so we claim a disturbed tab and no effect on outcomes. Two predictions and part of a third failed. Appendix M now opens with a plain list of all of them.
  • Significance. Where it occurs the error is large and silent: 127 reported passes against 22, with no error raised.

Response to Weakness 4, in detail (end of Section 4.4)

  • The method. Sort failures by a signature the intervention does not touch, then restore subsets of the overwritten state. It also held on 130 held-out trajectories.
  • Three observations. A later reader is needed: a call followed by at most two actions changed 0 of 208 replays. A partial fix can itself interfere. The observed error has one direction, so averaging does not remove it.

Response to Weakness 5, in detail

  • We claim no new isolation technique. Section 5 reports what a fork leaves open and our check of those routes over 5,375 calls.
  • The with-and-without test of Section 2.3 needs only two runs of a pipeline. It found problems with three different causes: a database, process state, and a browser tab.

We would be grateful if you would reconsider your scores in light of the rewritten paper.

Thank you for the clarifications.

  1. Could you confirm whether you've uploaded a revised manuscript to OpenReview? The "Revisions" page shows "No revisions to display" on my end. If a revised version is available, could you let me know so I can assess the changes?

  2. I've raised my soundness score. I still have concerns about whether the contribution is significant enough for publication at this venue, so I'm keeping my overall rating unchanged.

AuthorsOpenReview ↗

Follow-up to Reviewer 34Aw

Thank you for coming back to us, and for raising the soundness score.

The revised manuscript is uploaded

  • It replaced the original file, so it is the PDF linked at the top of this page (the PDF icon next to the title). We believe the "Revisions" page lists the edit history only for authors and chairs, which would explain why it is empty on your side.
  • The figures in it are redrawn. You have the revised file if Figure 2 on page 5 has four panels, (a) to (d), and Figure 1 on page 2 shows the lines of τ-bench's reward code. If an older file opens, a forced reload should fetch the new one.

On significance, your remaining concern. Where the revised paper addresses it:

  • Is it already fixed? In part, and the paper shows this. Figure 2(d) includes the patched evaluator that BrowserGym installs: it leaves the agent's tab alone, as you pointed out, while WebArena's original evaluator moves it (Section 3.3). Section 3.1 says that τ²-bench is not affected. AppWorld's own analysis script makes the extra calls during an episode (Section 3.2).
  • How large is the error, and would anyone see it? A public τ-bench training environment reports 127 passes on 159 episodes where the benchmark's own grade gives 22 (Figure 2(a)). On AppWorld, 17 of 142 published successes are lost when the grader is also called after every request (Figure 2(c)). Neither raises an error.
  • What carries over to other benchmarks? A test that needs two runs and no knowledge of the grader (Section 2.3), three conditions that say when a grading call is safe (Section 2.2), and three observations from AppWorld (end of Section 4.4).
  • What we do not claim. We do not measure how common such pipelines are, and we show no published number to be wrong (Section 7).

We would be grateful if you would look at the revised paper before you finalise your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.