- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces EnvOS, a protocol for auditing computer-use agent evaluations under controlled mid-task state changes. The protocol examines archived records to identify errors in the evaluation process that might otherwise be mistaken for agent failures. The results show that presenting current information immediately before submission can substantially improve success for some model–task configurations.
Critical strengths
The paper addresses a practical concern: apparent agent failures may originate from faulty interventions, incomplete observation delivery, or incorrect scoring. Distinguishing these sources is valuable for reliable agent evaluation.
Critical weaknesses
- The paper is difficult to follow. The early sections are overloaded with implementation details, protocol-specific terminology, and statistical qualifications, making the paper difficult to follow. The central idea and design rationale become obscured, and it is hard to understand how the individual components fit together. The paper needs a clearer high-level explanation of the problem. The writing should be improved.
- The prospective evaluation covers only two workflows sharing the same scaffold and interface. Both primarily involve a single authoritative value changing after observation. This provides limited evidence about applicability to repeated changes, conflicting sources, coupled decisions, or other environments.
Questions for authors
Both workflows use the same scaffold and involve a similar type of state change. What evidence suggests that EnvOS would also be useful for tasks with more complex changes, such as conflicting information sources?
3 comments
Further Clarifications on Experimental Scope and Generality
Dear Reviewer, Thank you again for your review and for raising concerns about the limited diversity of the prospective evaluation. We have posted detailed responses and substantially revised the manuscript. In particular, we added pre-registered experiments covering conflicting information sources and silently altered bookings, along with further scenarios involving competing users and post-confirmation cancellations. These experiments extend the evaluation beyond a single value changing after observation. We also rewrote the introduction and methodology to improve clarity and explicitly acknowledge the shared scaffold as a limitation.
We would appreciate it if you could revisit our responses and let us know whether these additions address your concerns. Thank you for your time and consideration.
Dear Authors,
Thank you for the revisions. The introduction is clearer, but I still find the manuscript difficult to follow. As you note in your AI Use Statement, you used AI tools to draft some sections, and I am concerned that this may have contributed to the lack of clarity. I believe substantial revision is needed before acceptance, so I will keep my original score.
Addressing Concerns on Presentation and Experimental Generality
Thank you for the review.
W1. The paper is hard to follow. We rewrote the paper around one plain question: when an agent fails after the world changes, was it the agent's fault?
W2 / Q. Beyond one kind of change: e.g. conflicting sources. We added two new pre-registered tasks, each run before and after the booking is submitted.
Conflicting sources. The session moves to another clinic. A pop-up summary either states the move, or wrongly says "No other changes". Only the portal's sessions page, outside the booking flow, shows the truth.
Stored differently. The portal silently saves a different time slot from the one submitted. Opus fixed it in 11/16 instances when this happened before submitting, but only 3/16 after (significant, p=0.043 after correction). It did report the wrong slot when the instruction asked it to (14/16). Sonnet 5 rarely did either: it fixed the slot in 5/16 before vs 0/16 after submitting, and reported it in 2/16 in each case.
Two further kinds of change, new since the revised PDF (not yet in it). These go beyond a single value changing after it was read: one adds a second actor competing for the same resource, the other a change that comes after the agent has already succeeded. Each was certified before any model ran and run on Opus 5 and Sonnet 5:
Each new task was certified before any model ran: scripted correct and deliberately flawed agents had to pass and fail as expected. Certification caught real faults in the conflicting-sources and stored-differently tasks before any model run. Each of those two needed only 6–9 new files, reusing the existing world, harness and scoring code. We also ran an open-weight model (Qwen3.6-35B-A3B) on the clinic task; it rarely completed it in any condition (at most 1/16). All tasks still share one scaffold, which we state as the main limit on generality.
Revised PDF. We have uploaded a revised PDF. It adds the conflicting-sources and stored-differently tasks, tests Sonnet 5 on them as well as Opus 5, and tests an open-weight model (Qwen3.6-35B-A3B) on the clinic task. The rewritten introduction and protocol are Sections 1 and 3, the overview of changes is Table 1, and the new tasks are in Sections 4, 5.3 and 5.4. The contested-slot and accepted-then-cancelled results are not yet in the PDF; we will add them in the next revision.