- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CLASHarness generates computer-use tasks in which simulated peer agents report conflicting information, and the agent has to check the real state, often from rendered screenshots, before acting. The pipeline fixes the decisive value before rendering, scans for text leaks, and derives code and LLM-judged checks from an executed reference trajectory. The benchmark has 600 rounds (20 seeds × 3 variants × 10 rounds). Across 15 models the best pass rate is 28.7%, and the paper adds cross-judge checks, a blind ablation, step-budget scaling, and a second benchmark built with another generator.
Critical strengths
-
The setting matters and is missing from current computer-use benchmarks. Several design choices, such as committing the answer before rendering, scanning for leaks, and compiling checks from executed trajectories, are useful well beyond this benchmark. This is the main reason I lean toward acceptance.
-
The evaluation is broad and yields practical findings, in particular the gap between per-check scores (55–67%) and full pass rates (at most 28.7%), and the consistent failure to complete closing actions such as releasing locks.
Critical weaknesses
-
No task has been checked by a human. Deriving checks from the reference trajectory guarantees that the reference passes the code checks, not that the instruction has a unique resolution or that the 65% LLM-judged criteria are fair. With pass rates below 30%, some failures may be task defects. I also think "multi-agent" oversells the setup: peers are static transcripts and the agent is told in advance that a contradiction exists, so the benchmark mainly tests resistance to misleading context and protocol compliance.
-
The statistical evidence is thinner than the presentation suggests. All 600 rounds come from only 20 seeds, so even with the seed-cluster bootstrap used for Table 1, the effective sample is small. Under that bootstrap, the resolve-versus-investigate gap is only marginal (p = 0.052) and compares non-randomized seed groups, yet it is presented as a main finding. The benchmark also depends heavily on its generator: Gemini 3.6 Flash is both the default generator and the default judge, and switching generators lowers every model's score.
Post-rebuttal update
The human audit found that 58 of 60 sampled rounds were solvable and 57 had a unique intended resolution. This substantially addresses my main concern about task validity. The matched 20-step result also addresses my budget concern. The use of static peer messages still limits the scope of the claims about multi-agent interaction. On balance, I’m raising my rating from 3 to 4.
Questions for authors
- Can you report a human audit of sampled rounds (solvability, uniqueness of the intended resolution, and reasonableness of each check)? This would likely raise my score.
- GPT-6 Astra, the top model in Table 1, uses an expanded budget (about 26 steps) while all other models use 20. How does it score at 20 steps?
- What exactly is the agent told about the contradiction, and how do results change if it is not told?
6 comments
Response to Reviewer eGsC (2/5)
W1b and Q3. Is the setting multi-agent, what exactly is the agent told about the conflict, and how do results change if it is not told?
Response. Yes, the setting is multi-agent: the evaluated agent has peer agents that answer it and do work for it. We answer the three parts in turn.
The setting is multi-agent. Each round has three to five peer agents. The evaluated agent messages them and delegates tasks to them. They reply, and they deliver their results as time passes, so the agent has to wait for them and check what they return. 598 of the 600 rounds grade the content of at least one message the agent must send. We script the peer agents by design: every model then faces exactly the same conflict and every result can be reproduced, while the evaluated agent still has other agents working with it. Sec. 3.1 now describes this design: scripted, reactive peer agents, chosen for control and reproducibility.
What the agent is told. The system prompt does not mention conflicts. It explains the tools (visual assets, wait, sub-agent outputs), tells the agent to verify the evidence itself, and tells it not to call done until the task's state changes have actually happened. Each instruction describes the peer agents' competing claims without saying which is correct, e.g. "Quinn_Engineering_Manager is urging an immediate grant … However, policy rules in the audit portal documentation dictate that a permit can only be issued if every required authority explicitly approves the exact requested target resource and scope." The agent must decide from the evidence which claim holds. While the agent has run only read-only commands, the harness reminds it that done is not yet allowed; it never says which state change is required. App. F.6 now prints the system prompt, an example instruction and this reminder.
If the agent is not told. We ran this ablation on 120 rounds. We rewrote each instruction to drop the description of the peer claims and kept the goal, deliverables and required actions. The workspace is unchanged, so the conflict can still be found in peer messages, logs and visual assets. All three models score lower (%):
| Model | Rounds passed, conflict described | Rounds passed, not described | Message content, described → not described |
|---|---|---|---|
| Claude Opus 5 | 20.8 | 14.2 | 49.8 → 42.5 |
| Gemini 3.6 Flash | 18.3 | 16.7 | 47.1 → 40.7 |
| GPT-6.1 Sol | 23.3 | 16.7 | 52.5 → 46.6 |
So finding the conflict without being told is a measurable part of the task. The models still open the required evidence and record the required values; what drops is the content of their messages and reports. The revision reports this ablation in Sec. 6 (Table 4b) and App. F.10 (Table F9).
What the benchmark tests. The conflict comes from the peer agents: their claims and reports disagree with each other or with the environment. The agent has to find out from the visual evidence which claim holds (without images, pass rates fall to 11–17% of their sighted values). It then has to act on the decision: message the peer agents and change the state. This is where most failures occur: 84–85% of reports state the correct decision, but models miss 74.8% of the closing actions that the instruction does not name.
Response to Reviewer eGsC (3/5)
W2a. All 600 rounds come from 20 seeds. Do the main conclusions hold when uncertainty is computed over seeds?
Response. Yes. A seed is a general scenario description. From each seed the generator samples three packages of ten rounds at temperature 0.4, and the rounds differ widely: 2,086 of the benchmark's 2,137 semantic criteria are distinct. Table 1 now reports seed-clustered standard errors, App. F.12 gives the pairwise tests, and the main conclusions hold under them.
GPT-6 Astra, Claude Opus 5 and GPT-5.6 Sol form the leading group. They are frontier models of the same generation, so close scores among them are expected. The benchmark separates this group from the next tier: Astra is significantly ahead of Claude Opus 4.8 and Claude Sonnet 5. Pass rate (%) ± seed-clustered SE, 600 rounds, 20 steps:
| Model | Pass rate |
|---|---|
| GPT-6 Astra | 27.5 ± 3.3 |
| Claude Opus 5 | 25.2 ± 2.5 |
| GPT-5.6 Sol | 24.5 ± 3.0 |
| Claude Opus 4.8 | 22.8 ± 2.6 |
| Kimi K3 | 21.8 ± 2.5 |
| Claude Sonnet 5 | 20.8 ± 3.0 |
Pairwise tests among these six models (Holm correction over 15 pairs):
| Pair | Δ (pp) | Seed-clustered 95% CI | Holm p, seed bootstrap |
|---|---|---|---|
| Astra vs. Claude Opus 5 | +2.3 | [−1.0, +5.8] | 1.0 |
| Astra vs. GPT-5.6 Sol | +3.0 | [−0.8, +6.5] | .98 |
| Claude Opus 5 vs. GPT-5.6 Sol | +0.7 | [−2.8, +4.2] | 1.0 |
| Astra vs. Claude Opus 4.8 | +4.7 | [+1.7, +7.5] | .022 |
| Astra vs. Claude Sonnet 5 | +6.7 | [+3.8, +9.8] | <.001 |
Two further results support the conclusions. The model ranking is reproduced on a corpus written by a different generator (ρ = 0.96). The new ablations are paired: they compare the same rounds with and without the intervention.
Response to Reviewer eGsC (1/5)
Thank you for the careful review and the concrete audit questions. We are encouraged that you find the setting important and missing from current computer-use benchmarks, and the design choices (committing the answer before rendering, leak scanning, checks compiled from executed trajectories) useful well beyond this benchmark. We also appreciate that you find the evaluation broad and its findings practical, in particular the gap between per-check scores and full pass rates and the missed closing actions. The new results below build on these strengths.
We follow the order of your review: comments W1 and W2 (with their sub-points), then questions. Where a question overlaps with a comment, we answer both together.
Response. Thank you for these concrete questions. The pipeline is built to settle them by construction: every round has an executed reference solution, one deciding fact fixed before rendering, and checks compiled from the execution. The revision adds a human audit organized by your three questions (summary in Table 4a, Sec. 6; full results in App. F.8). Three annotators independently audit 60 rounds (three per seed, one per difficulty level) and the 337 required checks of these rounds. They work from what the agent can see. Each result is the majority answer of the three annotators.
The annotators find 96.7% of the sampled tasks solvable, one intended resolution in 95.0%, and 96.1% of the checks reasonable.
The other evidence points the same way.
The LLM-judged criteria are graded consistently. Judges from the other two vendors re-graded all semantic checks of 14 models, and six additional models re-graded 170 judge verdicts from the audited rounds with the evaluated model hidden:
The main diagnostic findings (visual grounding, evidence citation, closing actions, deadlines) use deterministic checks only.
Every round has a verified solution. Each round has a reference trajectory, and its checks are compiled from its execution. When the corpus was built, the reference trajectory passed all of its round's checks in 600 of 600 rounds. With an exit checklist that names the required state changes, the number of rounds passed by at least one of three models rises from 72 to 147 of 264: the rounds can be solved once the closing actions are named.
One deciding fact per round. Each round fixes the single fact that settles its conflict before rendering, and the reference decision follows from that fact.
The difficulty is real. The low pass rates come from what the benchmark is built to measure. Models state the correct decision in most rounds (84–85% of reports on 120 rounds), but they miss 74.8% of the closing actions that the instruction does not name, against 9.2% of those it names. When a checklist names these actions, models perform 93–100% of them. The ranking does not depend on which closing-action checks are counted. Two LLM annotators (Gemini 4 Argon, Claude Opus 5.5) read only what the agent can see and mark the closing-action checks that have a basis in the task materials. With only these checks counted, no model passes more than 30.5% of rounds, and the same three models lead.