acceptodds
Under review as a conference paper at ICLR 2027

EnvOS: Auditing Computer-Use Agents When the World Moves On

Abstract

Computer-use agents act on information that can become stale before they commit, but apparent failures under such change can also be produced by the evaluation itself. We introduce EnvOS, a protocol for auditing controlled mid-task state changes: changes fire only after the soon-stale value was served; feasibility, corruptions, isolation and delivery are checked; rules are frozen before rollout; separate code re-derives validity verdicts and the outcome terms supported by archived evidence. We evaluate the finalized protocol prospectively in two workflows under one controlled semantic-browser interface. In clinic booking, Claude Opus 5 succeeded on 0/16 instances with no panel, 2/16 with a cue-and-link panel and 15/16 when the same panel displayed the current location (LIVE–CUE: 13 rescues, 0 harms and 3 ties, block-local Holm p=0.000488; CUE–OFF: block-local Holm p=0.5). At the registered block-local level, displaying the current location increased success relative to this cue-and-link panel. The CUE–OFF contrast did not reject; this does not establish absence of a cue effect. Contradiction, salience, and reduced verification effort remain possible explanations. Over all twelve claim-bearing tests, LIVE–CUE's Holm p=0.00293. In insurance-claim approval, Claude Opus 5 succeeded on 10/10 silent-change instances without review-step re-exposure, Claude Sonnet 5 on 1/10 without and 9/10 with it (8 rescues, 0 harms, 2 ties; registered block-local p=0.00781; retrospective twelve-test Holm p=0.0781). A separately pre-registered follow-up with a non-Anthropic configuration (Gemini 3.1 Pro Preview) did not meet the registered effect criterion (0/10 to 5/10, exact p = 0.0625). Evaluator audits exposed their own limits: on a sealed challenge, validity re-derivation met 24/28 expectations; on a second, specified without implementation or outcome access, the pipeline counted none of 26 corruptions as clean but gave no verdict on 2; on a third, which an external agent designed and generated, it caught more corruptions than raw-log baselines in both workflows. It still counted 12 of 39 as clean. These contrasts estimate within-configuration effects; unrejected comparisons do not establish absence or equivalence.

Primary area:
reinforcement learning
Keywords:
computer-use agents, agent evaluation, evaluation validity, pre-registration, mid-task state changes, causal evaluation, auditing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EGRDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces EnvOS, a protocol for auditing computer-use agent evaluations under controlled mid-task state changes. The protocol examines archived records to identify errors in the evaluation process that might otherwise be mistaken for agent failures. The results show that presenting current information immediately before submission can substantially improve success for some model–task configurations.

Critical strengths

The paper addresses a practical concern: apparent agent failures may originate from faulty interventions, incomplete observation delivery, or incorrect scoring. Distinguishing these sources is valuable for reliable agent evaluation.

Critical weaknesses
  1. The paper is difficult to follow. The early sections are overloaded with implementation details, protocol-specific terminology, and statistical qualifications, making the paper difficult to follow. The central idea and design rationale become obscured, and it is hard to understand how the individual components fit together. The paper needs a clearer high-level explanation of the problem. The writing should be improved.
  2. The prospective evaluation covers only two workflows sharing the same scaffold and interface. Both primarily involve a single authoritative value changing after observation. This provides limited evidence about applicability to repeated changes, conflicting sources, coupled decisions, or other environments.
Questions for authors

Both workflows use the same scaffold and involve a similar type of state change. What evidence suggests that EnvOS would also be useful for tasks with more complex changes, such as conflicting information sources?

3 comments
AuthorsOpenReview ↗

Addressing Concerns on Presentation and Experimental Generality

Thank you for the review.

W1. The paper is hard to follow. We rewrote the paper around one plain question: when an agent fails after the world changes, was it the agent's fault?

  • The introduction starts from one concrete case. A booking agent seems to use an outdated clinic location, but the change may have fired too early, never reached the agent, or been misjudged by the scorer.
  • The protocol section explains, in that order:
    • what changes;
    • when it fires;
    • how an episode is scored;
    • how each test is checked before any model runs.
  • Codenames, amendment numbers and implementation details are removed from the main text. Statistics are explained once, briefly.
  • One table lists every kind of change tested, and one table and one figure give every result. Each result section asks one question and states one takeaway.

W2 / Q. Beyond one kind of change: e.g. conflicting sources. We added two new pre-registered tasks, each run before and after the booking is submitted.

Conflicting sources. The session moves to another clinic. A pop-up summary either states the move, or wrongly says "No other changes". Only the portal's sessions page, outside the booking flow, shows the truth.

  • Opus 5, before booking: 16/16 correct with either summary. It checked the sessions page even when told "No other changes", and twice called the summary a possible prompt injection.
  • Opus 5, after booking: 16/16 with the correct summary vs 11/16 with the misleading one. Every instance that differed favoured the correct summary, but 5 such instances are too few for significance (p=0.0625p = 0.0625). This is the pre-registered follow-up at 64 actions; at the first budget of 48, it was 10/16 vs 7/16.
  • Sonnet 5, before booking: 15/16 vs 11/16 (not significant, p=0.219p = 0.219); 5 instances favoured the correct summary, 1 the misleading one.

Stored differently. The portal silently saves a different time slot from the one submitted. Opus fixed it in 11/16 instances when this happened before submitting, but only 3/16 after (significant, p=0.043p = 0.043 after correction). It did report the wrong slot when the instruction asked it to (14/16). Sonnet 5 rarely did either: it fixed the slot in 5/16 before vs 0/16 after submitting, and reported it in 2/16 in each case.

Two further kinds of change, new since the revised PDF (not yet in it). These go beyond a single value changing after it was read: one adds a second actor competing for the same resource, the other a change that comes after the agent has already succeeded. Each was certified before any model ran and run on Opus 5 and Sonnet 5:

  • Contested slot (competing users). While the agent books 15:00, another patient takes it, either before the agent submits or as it submits. The portal either says the slot is no longer available, or shows only a status word ("full", "waitlisted"). Opus: 16/16 vs 16/16 before submitting, 16/16 vs 13/16 as it submits. Sonnet: 14/16 vs 16/16, and 15/16 vs 12/16. No difference is significant; both models handle this well overall. Of Opus's 3 failures with only the status word, 2 kept the waitlisted booking and mentioned the problem; 1 ran out of actions.
  • Accepted, then cancelled (a change after success). The booking is saved and shown to the agent as confirmed; right after, the clinic cancels it. Either a message on every later page says so, or only the booking's status changes. Both models succeeded in 0/16 in both conditions, and in 3/3 controls without the cancellation. Most runs ended without ever seeing the cancellation (Opus 28/32, Sonnet 29/32). When Opus did notice (3/16 with the message, 1/16 without), it booked again but always at the next slot, 15:30, apparently reading the cancellation as "15:00 is unavailable"; we count these as failures, as registered. Sonnet noticed 3 times and ran out of actions each time. The agents' failure here is clear; which signal helps more cannot be told at this floor.

Each new task was certified before any model ran: scripted correct and deliberately flawed agents had to pass and fail as expected. Certification caught real faults in the conflicting-sources and stored-differently tasks before any model run. Each of those two needed only 6–9 new files, reusing the existing world, harness and scoring code. We also ran an open-weight model (Qwen3.6-35B-A3B) on the clinic task; it rarely completed it in any condition (at most 1/16). All tasks still share one scaffold, which we state as the main limit on generality.

Revised PDF. We have uploaded a revised PDF. It adds the conflicting-sources and stored-differently tasks, tests Sonnet 5 on them as well as Opus 5, and tests an open-weight model (Qwen3.6-35B-A3B) on the clinic task. The rewritten introduction and protocol are Sections 1 and 3, the overview of changes is Table 1, and the new tasks are in Sections 4, 5.3 and 5.4. The contested-slot and accepted-then-cancelled results are not yet in the PDF; we will add them in the next revision.

AuthorsOpenReview ↗

Further Clarifications on Experimental Scope and Generality

Dear Reviewer, Thank you again for your review and for raising concerns about the limited diversity of the prospective evaluation. We have posted detailed responses and substantially revised the manuscript. In particular, we added pre-registered experiments covering conflicting information sources and silently altered bookings, along with further scenarios involving competing users and post-confirmation cancellations. These experiments extend the evaluation beyond a single value changing after observation. We also rewrote the introduction and methodology to improve clarity and explicitly acknowledge the shared scaffold as a limitation.

We would appreciate it if you could revisit our responses and let us know whether these additions address your concerns. Thank you for your time and consideration.

Dear Authors,

Thank you for the revisions. The introduction is clearer, but I still find the manuscript difficult to follow. As you note in your AI Use Statement, you used AI tools to draft some sections, and I am concerned that this may have contributed to the lack of clarity. I believe substantial revision is needed before acceptance, so I will keep my original score.

Reviewer_zLmbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

EnvOS is a bounded protocol for auditing controlled mid-task changes in agent evaluations, combining exposure-conditioned authoritative state mutation, observation-delivery checks, validity-first counting, frozen interpretations and separate code paths for re-deriving supported verdict terms (pp. 2–4, Figure 1/sec. 3). Two prospective workflows share one semantic-browser harness. Clinic Opus 5 succeeds on 0/16 no-panel, 2/16 cue-and-link and 15/16 current-value instances; LIVE–CUE has registered block-Holm p = 0.000488 and reported paper-wide sensitivity p = 0.00293. Independent reconstruction of the clinic full endpoint remains conditional on calendar compatibility (pp. 6–7, Table 1/sec. 5.2; p. 20, Table 9). Claims Sonnet 5 improves from 1/10 to 9/10 with the panel (registered local p = 0.00781; retrospective paper-wide p = 0.0781); Opus succeeds 10/10 in both arms, while Gemini's 0/10 versus 5/10 does not meet its registered criterion (pp. 6–7). Three archive challenges show added checking coverage alongside concrete timing and copy-consistency gaps (pp. 7–8; pp. 31–32, Appendix F). These are fixed-instance, single-rollout panel effects with change present in both arms; they neither isolate a unique panel mechanism nor estimate the effect of environmental change itself. Results are paper-reported, not reproduced here. (pp. 4–5, 9.)

Critical strengths
  1. An operational decomposition of evaluation validity. Exposure-conditioned changes, separately recorded delivery, validity-first counting and separate outcome/validity implementations make plausible evaluator failures testable (pp. 3–4, sec. 3). The challenges and 11 reported development defects show that checking layers can disagree; the authors acknowledge shared evidence and known-fault-class limits (pp. 7–9). This gives the protocol methodological value beyond a new task score. Independent here means separate code, not an independent team or independent evidence.

  2. A controlled and carefully qualified panel comparison. Clinic CUE/LIVE differs in the location field, with 13 treatment-only successes, no opposite discordances and 3 ties; its reported evaluator-endpoint contrast survives both Holm analyses. Independent verification of full clinic success remains conditional on calendar compatibility (p. 5 Figure 2; p. 7 sec. 5.2; p. 20 Table 9). Claims adds a derived decision rather than displaying the answer, and Sonnet's result is positive under its preregistered local test. The retrospective paper-wide non-rejection limits broader inferential claims without invalidating that registered result. Opus's ceiling, Gemini's non-rejection and the incomplete clinic replication are reported transparently. These are bounded within-configuration findings, not evidence of vendor or environment generality. (pp. 4–7.)

Critical weaknesses
  1. Concrete coverage gaps limit the protocol's assurance. C08 (commit before change) enters unflagged under both clinic readings; the implemented reading also misses delivery corruptions C01/C02 (p. 8 Table 2). Appendix F identifies undisputed clean-counted gaps EB14 (collateral-state/status disagreement) and EB18 (inconsistent event-version copies). These concern central timing and consistency properties, although they do not prove any actual BC episode was corrupted or establish equivalence of every prospective operational path. Authors dispute ten of twelve clean-counted external expectations and all three rejected-control expectations; those disputes do not eliminate EB14/EB18 and were not independently adjudicated here (pp. 31–32). Clinic full-endpoint assurance also retains the explicit calendar assumption, not established by the inspected neutral-calendar checks (p. 20 Table 9; p. 52 targeted paragraphs).

  2. The incremental assurance demonstrated over simpler checks is limited. Within the simple baseline's declared scope, the clinic protocol and baseline both reject 11/12 corruptions; claims improves from 6/12 to 10/12, a real positive result. The broader totals partly compare different coverage, and the comparison is post hoc and author-built (p. 8 Table 2/sec. 6). The external challenge provides independently constructed cases but uses one base per workflow; the authors acknowledge narrower baseline scope and untested coverage-matched alternatives (pp. 31–32). Thus the evidence supports useful per-case additions, not a general detection-rate or architectural-superiority claim. Combined with one shared harness and single rollouts, this limits how much practical benefit of the full protocol I can credit. Broader environments or repeated sampling would be extensions, not requirements for its stated bounded result. (pp. 4–5, 8–9.)

Questions for authors
  1. Which frozen version and checks protect the prospective counting path against C08 and EB14/EB18, and were their corresponding properties checked on the actual counted episodes? Please distinguish constructed-case failures from evidence of any affected prospective result. This would determine how much operational assurance to credit; it does not assume the episodes are faulty. (p. 8 Table 2; p. 31 Appendix F.)

  2. Which existing checks establish calendar compatibility for BC's committed slots, and what conclusion about full-task success remains if that term is unresolved? Table 9 retains the assumption; the p. 52 paragraphs establish equal neutral work and absence of clinic cues, rather than independently reconstructing slot compatibility. This matters to the full-endpoint interpretation of the strongest panel result. No new environment or broad experimental suite is requested. (p. 7 sec. 5.2; p. 20 Table 9; p. 52 targeted passages.)

Reviewer_EeCcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EnvOS is a protocol for attributing a computer-use agent’s failure to a mid-task record change rather than to harness or scoring faults. The change fires only after the agent was served the old value, a control applies the same trigger to an irrelevant record, scripted policies certify every condition, and challenge sets audit the evaluator.

Critical strengths

The attribution design is useful to benchmark builders. OSWorld 2.0 and STT-Arena ask whether agents fail under change; EnvOS’s trigger, control, certification and pre-registration make it checkable whether a failure is due to the change.

The evaluator audits are fairly good evidence in their own right. For 7 of 11 development defects, archived rows already carried model verdicts, e.g. a provider session-limit reply scored as a model failure (Section 6, Table 34).

Critical weaknesses

The model findings are narrow and quite close to handing the agent the answer. In clinic booking, the value panel puts the current location beside the stale draft (Section 4, Figure 2), so its 13 rescues, 0 harms and 3 ties over the link panel mainly show Opus acting on a contradiction placed before it; the link was followed in 1/16 runs (Section 5.2). In claims, a scripted policy that copies the displayed limit already gets 7 rescues, 0 harms and 3 ties; only 3 threshold-crossing pairs separate Sonnet from it (Appendix H). Gemini’s five panel failures all ran out of steps without a stale approval (Table 20).

Generality and ease of adoption are not adequately shown. Both workflows share one self-built scaffold and harness, and all Claude runs used the Claude Code CLI, whose system prompt was not archived (Section 7, Appendix R). The paper claims only task-level reuse, but nothing shows which checks are necessary or what porting costs. The text reads as an audit log: about twenty block codenames, 19 addenda, and one template sentence repeated verbatim in four places.

Questions for authors

(Clarification) Does the clinic link panel contain any sentence prompting a check? Please print it, as Appendix R(6) does for the earlier value panel.

(Suggestion) A panel stating that the location changed, without the new value, would hel[ separate signalling the change from supplying the answer.

(Clarification) Appendix K has templates for reuse counts but reports no values. What did the claims task add to the clinic scaffold?

2 comments
AuthorsOpenReview ↗

Addressing Concerns on Answer Leakage and Generality

Thank you for the careful review and the concrete suggestion.

W1. The panel result may mostly hand the agent the answer. We agree that comparing the value panel with the link panel could not tell signalling that the location changed apart from giving the new location. Following your suggestion, we ran a new pre-registered block on 16 new clinic instances with three panels:

  • the link panel;
  • the link panel plus one sentence, "The location for this session has changed." (no new location given);
  • the value panel.

Opus 5 succeeded in 0/16 with the link panel, 16/16 with the added sentence and 16/16 with the value panel. Sentence vs link: all 16 instances improved, none got worse (exact sign test, p=6.1×10−5p = 6.1\times10^{-5}, after Holm correction for the block's two tests).

So telling the agent that the location changed was enough; it then looked up the new location itself. Being handed the value is not needed to explain the original result. The sentence panel still contains the link, so we do not claim the sentence works without it.

Other models. Sonnet 5, same instances: link 0/16, sentence 14/16, value 13/16 (descriptive: our second scorer could not confirm one term on 3 runs). Qwen3.6-35B-A3B, 16 fresh instances: 0/16, 1/16, 0/16; the change often never fired (27 of 51 runs), and such runs count as failures.

Claims. We agree that a script copying the displayed limit would also fix 7 of the 10 pairs (Sonnet fixed 8). The pre-registered check targets the 3 instances where copying gives the wrong amount; Sonnet with the panel was correct on all 3, and that is the only part of the claims result we read as more than copying.

Gemini. All five failures with the panel ran out of actions without approving a stale amount. The block did not meet its registered criterion, and we report it only as that.

W2. Generality, cost, necessary checks, readability.

  • More kinds of change. Four new pre-registered tasks with different failure modes, each certified before any model ran (scripted correct and deliberately flawed agents had to pass and fail as expected):

    • the portal saves a different slot from the one submitted;
    • a posted summary contradicts the authoritative page (both run before and after the booking is submitted);
    • new since the revised PDF: another patient takes the chosen slot during booking (the portal says so, or shows only "full"/"waitlisted"): Opus 16/16 vs 16/16 before submitting, 16/16 vs 13/16 as it submits; Sonnet 14/16 vs 16/16, 15/16 vs 12/16; no difference is significant, and both models are near ceiling;
    • new since the revised PDF: the booking is shown as confirmed, then the clinic cancels it (a message says so, or only the status changes): Opus and Sonnet 0/16 in both conditions, 3/3 without the cancellation; most runs replied right after seeing "confirmed" and never looked again (Opus 28/32, Sonnet 29/32). With no successes, the two signals cannot be compared; the failure itself is the finding.

    We also added a self-served open-weight model (Qwen3.6-35B-A3B). All tasks still share one scaffold, and we state this as the main limit.

  • Which checks are necessary. Removing one audit layer at a time (post hoc): corruptions caught on the sealed clinic set fall from 16/18 to 7/18 without the validity re-derivation; claims from 19/20 to 14/20 without the harness's own verdict.

  • System prompt. The Claude CLI's built-in system prompt is still not archived, and we say so. For the open-weight model (Qwen3.6-35B-A3B, run on our own GPU), no CLI is involved: our own code sends every request, with fixed model weights and decoding settings.

  • Readability. Block codenames, addendum numbers and the repeated template sentence are removed from the main text.

Q1. Text of the link panel, as rendered in every counted run:

Session details for the selected date
Session: Influenza, [date]
Location: ————————————————
[Open the Clinic sessions page]

It contains no sentence prompting a check. Certification rejects a link or value panel containing any word from a frozen list of change and checking stems (e.g. "chang", "check", "verif", "please"); only the sentence panel's added sentence may use them.

Q2. Run as suggested, on 16 new instances; see W1.

Q3. Cost of reuse (costs, not evidence of portability).

  • Claims: 28 files (10,187 lines); 12 of 23 shared components byte-identical, the others changed by at most 1,357 lines.
  • Slot task: 9 files (2,821 lines with tests), +567 lines in 6 shared files.
  • Summary task: 6 files (1,649 lines), +403 lines in 5 shared files.

Revised PDF. We have uploaded a revised PDF. It adds the slot and summary tasks, also runs Sonnet 5 on them, and adds Qwen3.6-35B-A3B. Sentence block: Section 5.1, Table 2; new tasks: Sections 4, 5.3–5.4; layer removal: Section 6; panel text and reuse costs: Appendices G.1 and Q. The two tasks marked new are not yet in the PDF; we will add them in the next revision.

AuthorsOpenReview ↗

Additional Experiments and Clarifications on Generality

Dear Reviewer, Thank you again for your careful review and constructive suggestions. We have posted detailed responses addressing your concerns and uploaded a substantially revised manuscript. In particular, we added the suggested experiment separating change signalling from providing the updated value (Opus: 0/16 vs. 16/16), introduced additional tasks covering conflicting sources and other environmental changes, and improved the presentation and clarified the limitations.

We would appreciate it if you could revisit our responses and let us know whether these revisions address your concerns. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.