acceptodds
Under review as a conference paper at ICLR 2027

CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline

Abstract

Vision-language models (VLMs) are increasingly deployed in multi-agent systems to automate complex computer-use workflows, prompting numerous benchmarks to evaluate and advance agent capabilities. However, existing benchmarks mostly assume consistent, cooperative environments. In contrast, real-world collaboration rarely runs on consistent information: peer or sub-agents frequently report outdated states, issue contradictory instructions, or compete over shared permissions and files, forcing an agent to check others' outputs and verify ground truth in the operating environment. Evaluating such conflict resolution remains challenging: manual task authoring cannot scale across complex systems, while generating tasks via unconstrained LLM generation produces narrative drift, answer leakage, and unsatisfiable grading rubrics that leave tasks unsolvable. We introduce CLASHarness (Conflict-Loaded Agent Scenario Harness), a controllable framework for generating verifiable multi-agent conflicts in computer use, accompanied by a benchmark of 600 multi-round tasks. By combining hierarchical scenario planning with an eight-step verified micro-pipeline, CLASHarness pre-commits decisive visual facts before rendering and compiles hybrid code and semantic graders directly from executed reference trajectories, guaranteeing that every task is solvable by construction. Evaluating 15 frontier models on CLASHarness reveals three core bottlenecks. First, when outputs of sub-agents conflict with the environment, weaker models blindly follow misleading information, whereas stronger models will inspect visual evidence to verify ground truth. Second, resolving conflicts is much harder than finding them: while agents can spot inconsistencies, actively fixing the system state drops task success significantly. Third, coordination protocols remain a universal bottleneck: even frontier models fail over 58% of closing actions, frequently forgetting to release shared locks or hand off tasks to sub-agents. CLASHarness provides a scalable, verifiable foundation for diagnosing coordination failures and building reliable multi-agent systems. Code and Dataset will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
Computer-Use Agents, Multi-Agent Systems, Conflict Resolution, Benchmarks, Vision-Language Models, Procedural Task Generation, Tool Use

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_eGsCOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CLASHarness generates computer-use tasks in which simulated peer agents report conflicting information, and the agent has to check the real state, often from rendered screenshots, before acting. The pipeline fixes the decisive value before rendering, scans for text leaks, and derives code and LLM-judged checks from an executed reference trajectory. The benchmark has 600 rounds (20 seeds × 3 variants × 10 rounds). Across 15 models the best pass rate is 28.7%, and the paper adds cross-judge checks, a blind ablation, step-budget scaling, and a second benchmark built with another generator.

Critical strengths
  1. The setting matters and is missing from current computer-use benchmarks. Several design choices, such as committing the answer before rendering, scanning for leaks, and compiling checks from executed trajectories, are useful well beyond this benchmark. This is the main reason I lean toward acceptance.

  2. The evaluation is broad and yields practical findings, in particular the gap between per-check scores (55–67%) and full pass rates (at most 28.7%), and the consistent failure to complete closing actions such as releasing locks.

Critical weaknesses
  1. No task has been checked by a human. Deriving checks from the reference trajectory guarantees that the reference passes the code checks, not that the instruction has a unique resolution or that the 65% LLM-judged criteria are fair. With pass rates below 30%, some failures may be task defects. I also think "multi-agent" oversells the setup: peers are static transcripts and the agent is told in advance that a contradiction exists, so the benchmark mainly tests resistance to misleading context and protocol compliance.

  2. The statistical evidence is thinner than the presentation suggests. All 600 rounds come from only 20 seeds, so even with the seed-cluster bootstrap used for Table 1, the effective sample is small. Under that bootstrap, the resolve-versus-investigate gap is only marginal (p = 0.052) and compares non-randomized seed groups, yet it is presented as a main finding. The benchmark also depends heavily on its generator: Gemini 3.6 Flash is both the default generator and the default judge, and switching generators lowers every model's score.


Post-rebuttal update

The human audit found that 58 of 60 sampled rounds were solvable and 57 had a unique intended resolution. This substantially addresses my main concern about task validity. The matched 20-step result also addresses my budget concern. The use of static peer messages still limits the scope of the claims about multi-agent interaction. On balance, I’m raising my rating from 3 to 4.

Questions for authors
  1. Can you report a human audit of sampled rounds (solvability, uniqueness of the intended resolution, and reasonableness of each check)? This would likely raise my score.
  2. GPT-6 Astra, the top model in Table 1, uses an expanded budget (about 26 steps) while all other models use 20. How does it score at 20 steps?
  3. What exactly is the agent told about the contradiction, and how do results change if it is not told?
6 comments
AuthorsOpenReview ↗

Response to Reviewer eGsC (1/5)

Thank you for the careful review and the concrete audit questions. We are encouraged that you find the setting important and missing from current computer-use benchmarks, and the design choices (committing the answer before rendering, leak scanning, checks compiled from executed trajectories) useful well beyond this benchmark. We also appreciate that you find the evaluation broad and its findings practical, in particular the gap between per-check scores and full pass rates and the missed closing actions. The new results below build on these strengths.

We follow the order of your review: comments W1 and W2 (with their sub-points), then questions. Where a question overlaps with a comment, we answer both together.

W1a and Q1. Does a human check confirm that the tasks are solvable, that the intended resolution is unique, and that each check is reasonable? Do the low pass rates reflect real difficulty?

Response. Thank you for these concrete questions. The pipeline is built to settle them by construction: every round has an executed reference solution, one deciding fact fixed before rendering, and checks compiled from the execution. The revision adds a human audit organized by your three questions (summary in Table 4a, Sec. 6; full results in App. F.8). Three annotators independently audit 60 rounds (three per seed, one per difficulty level) and the 337 required checks of these rounds. They work from what the agent can see. Each result is the majority answer of the three annotators.

Your questionItemsResult (majority of three)Agreement among annotators
(1) Task is solvable60 rounds96.7% (58/60)92.2%
(2) Intended resolution is unique60 rounds95.0% (57/60)87.8%
(3) Check is reasonable337 checks96.1% (324/337)91.2%
(3) A reasonable alternative would fail the check337 checks5.0% (17/337)90.3%

The annotators find 96.7% of the sampled tasks solvable, one intended resolution in 95.0%, and 96.1% of the checks reasonable.

The other evidence points the same way.

The LLM-judged criteria are graded consistently. Judges from the other two vendors re-graded all semantic checks of 14 models, and six additional models re-graded 170 judge verdicts from the audited rounds with the evaluated model hidden:

Check on the judgeAgreement with the recorded grades
Full re-grade, Claude Sonnet 5κ = 0.922
Full re-grade, Claude Opus 5κ = 0.902
Full re-grade, GPT-6.1 Solκ = 0.921
170 verdicts, majority of six additional models (GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5, Claude Fable 5.1, Gemini 4 Argon, Gemini 4 Carbon)98.2%, κ = 0.96 (three ties not counted)

The main diagnostic findings (visual grounding, evidence citation, closing actions, deadlines) use deterministic checks only.

Every round has a verified solution. Each round has a reference trajectory, and its checks are compiled from its execution. When the corpus was built, the reference trajectory passed all of its round's checks in 600 of 600 rounds. With an exit checklist that names the required state changes, the number of rounds passed by at least one of three models rises from 72 to 147 of 264: the rounds can be solved once the closing actions are named.

One deciding fact per round. Each round fixes the single fact that settles its conflict before rendering, and the reference decision follows from that fact.

The difficulty is real. The low pass rates come from what the benchmark is built to measure. Models state the correct decision in most rounds (84–85% of reports on 120 rounds), but they miss 74.8% of the closing actions that the instruction does not name, against 9.2% of those it names. When a checklist names these actions, models perform 93–100% of them. The ranking does not depend on which closing-action checks are counted. Two LLM annotators (Gemini 4 Argon, Claude Opus 5.5) read only what the agent can see and mark the closing-action checks that have a basis in the task materials. With only these checks counted, no model passes more than 30.5% of rounds, and the same three models lead.

AuthorsOpenReview ↗

Response to Reviewer eGsC (2/5)

W1b and Q3. Is the setting multi-agent, what exactly is the agent told about the conflict, and how do results change if it is not told?

Response. Yes, the setting is multi-agent: the evaluated agent has peer agents that answer it and do work for it. We answer the three parts in turn.

The setting is multi-agent. Each round has three to five peer agents. The evaluated agent messages them and delegates tasks to them. They reply, and they deliver their results as time passes, so the agent has to wait for them and check what they return. 598 of the 600 rounds grade the content of at least one message the agent must send. We script the peer agents by design: every model then faces exactly the same conflict and every result can be reproduced, while the evaluated agent still has other agents working with it. Sec. 3.1 now describes this design: scripted, reactive peer agents, chosen for control and reproducibility.

What the agent is told. The system prompt does not mention conflicts. It explains the tools (visual assets, wait, sub-agent outputs), tells the agent to verify the evidence itself, and tells it not to call done until the task's state changes have actually happened. Each instruction describes the peer agents' competing claims without saying which is correct, e.g. "Quinn_Engineering_Manager is urging an immediate grant … However, policy rules in the audit portal documentation dictate that a permit can only be issued if every required authority explicitly approves the exact requested target resource and scope." The agent must decide from the evidence which claim holds. While the agent has run only read-only commands, the harness reminds it that done is not yet allowed; it never says which state change is required. App. F.6 now prints the system prompt, an example instruction and this reminder.

If the agent is not told. We ran this ablation on 120 rounds. We rewrote each instruction to drop the description of the peer claims and kept the goal, deliverables and required actions. The workspace is unchanged, so the conflict can still be found in peer messages, logs and visual assets. All three models score lower (%):

ModelRounds passed, conflict describedRounds passed, not describedMessage content, described → not described
Claude Opus 520.814.249.8 → 42.5
Gemini 3.6 Flash18.316.747.1 → 40.7
GPT-6.1 Sol23.316.752.5 → 46.6

So finding the conflict without being told is a measurable part of the task. The models still open the required evidence and record the required values; what drops is the content of their messages and reports. The revision reports this ablation in Sec. 6 (Table 4b) and App. F.10 (Table F9).

What the benchmark tests. The conflict comes from the peer agents: their claims and reports disagree with each other or with the environment. The agent has to find out from the visual evidence which claim holds (without images, pass rates fall to 11–17% of their sighted values). It then has to act on the decision: message the peer agents and change the state. This is where most failures occur: 84–85% of reports state the correct decision, but models miss 74.8% of the closing actions that the instruction does not name.

AuthorsOpenReview ↗

Response to Reviewer eGsC (3/5)

W2a. All 600 rounds come from 20 seeds. Do the main conclusions hold when uncertainty is computed over seeds?

Response. Yes. A seed is a general scenario description. From each seed the generator samples three packages of ten rounds at temperature 0.4, and the rounds differ widely: 2,086 of the benchmark's 2,137 semantic criteria are distinct. Table 1 now reports seed-clustered standard errors, App. F.12 gives the pairwise tests, and the main conclusions hold under them.

GPT-6 Astra, Claude Opus 5 and GPT-5.6 Sol form the leading group. They are frontier models of the same generation, so close scores among them are expected. The benchmark separates this group from the next tier: Astra is significantly ahead of Claude Opus 4.8 and Claude Sonnet 5. Pass rate (%) ± seed-clustered SE, 600 rounds, 20 steps:

ModelPass rate
GPT-6 Astra27.5 ± 3.3
Claude Opus 525.2 ± 2.5
GPT-5.6 Sol24.5 ± 3.0
Claude Opus 4.822.8 ± 2.6
Kimi K321.8 ± 2.5
Claude Sonnet 520.8 ± 3.0

Pairwise tests among these six models (Holm correction over 15 pairs):

PairΔ (pp)Seed-clustered 95% CIHolm p, seed bootstrap
Astra vs. Claude Opus 5+2.3[−1.0, +5.8]1.0
Astra vs. GPT-5.6 Sol+3.0[−0.8, +6.5].98
Claude Opus 5 vs. GPT-5.6 Sol+0.7[−2.8, +4.2]1.0
Astra vs. Claude Opus 4.8+4.7[+1.7, +7.5].022
Astra vs. Claude Sonnet 5+6.7[+3.8, +9.8]<.001

Two further results support the conclusions. The model ranking is reproduced on a corpus written by a different generator (ρ = 0.96). The new ablations are paired: they compare the same rounds with and without the intervention.

Reviewer_iuQeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission tries to answer this question: can an AI agent complete a computer task when other agents give it conflicting or outdated information? To address this, the authors introduced CLASHarness, a framework for generating and evaluating computer-use tasks in which an agent must reconcile conflicting peer reports, visual evidence, and shared system state. The benchmark comprises 60 scenario packages with 10 rounds each, totaling 600 conflict rounds across 20 operational domains. Its main contribution is a generation pipeline that combines hierarchical scenario planning, pre-committed visual facts, and grading checks derived from executed reference trajectories. This extends the benchmarks discussed in the paper by evaluating conflict handling through both evidence assessment and operational actions. Experiments across 15 models reveal substantial difficulty completing these workflows, particularly when agents must correct system state, release shared resources, or complete coordination protocols.

Critical strengths
  1. The proposed pipeline addresses an important source of benchmark unreliability: disagreement between task instructions, available evidence, and grading criteria. Figure 1 and Section 3 describe how scenario planning, pre-committed facts, reference execution, and checker compilation fit together. Grounding the rubric in an executed trajectory is a useful contribution beyond generating plausible task descriptions with an LLM.
  2. The paper provides convincing evidence that the visual inputs matter. Figure 2b shows a substantial performance decline when images are withheld. Importantly, this decline persists on tasks where image inspection is not explicitly enforced by the rubric: Opus 5 drops from 24.8% to 2.8%. This helps establish that the benchmark requires information from images rather than merely rewarding agents for opening them.
Critical weaknesses
  1. Table 1 presents GPT-6 Astra's 28.7% success rate alongside models evaluated with a smaller action allowance: Astra receives 40 steps, whereas the other models receive 20 steps, as specified in Appendix Table D1. This matters because Section 6, Figure 2(a) demonstrates substantial budget sensitivity. For example, Gemini 3.7 Flash improves from 20.3% at 20 steps to 27.2% at 40 steps, approaching Astra’s reported result. Please provide a common-budget comparison in Table 1, or show success-versus-budget curves for the leading models. This would better support the interpretation in Section 5.1 that performance differences reflect stronger handling of complex conflict trajectories, rather than differences in available exploration.

  2. In Section 5.3 and Table 2, the authors explicitly claim that “an early mistake causes the entire task to fail” on short tasks, whereas failures on long tasks arise from gradual mistakes. Their evidence is co-failure lift, which measures whether checks tend to fail together. But checks failing together does not show which mistake happened first or whether it caused subsequent mistakes. Several checks could, for example, evaluate the same incorrect output, producing clustered failures without a chain of execution errors. Please consider tracing the first error and its downstream consequences, or describing the result as clustered versus less-clustered failures rather than cascading versus gradual errors.

Questions for authors
  1. Could you address weakness 1?
  2. Could you address weakness 2?
  3. I suspect significant use of AI tools in presenting the results, including quite obvious patterns in table making. Can you justify that level of involvement?
4 comments
AuthorsOpenReview ↗

Response to Reviewer iuQe (1/2)

Thank you for the clear and actionable review. We appreciate that you see the pipeline as addressing an important source of benchmark unreliability, and that you find grounding the rubric in an executed trajectory a useful contribution. We are also glad that you find the evidence for the visual inputs convincing, including on tasks whose rubric does not enforce opening the images. Both of your suggestions are now in the revision.

We follow the order of your review. Your Q1 and Q2 ask us to address W1 and W2, so we answer them together.

W1 and Q1. Table 1 gives GPT-6 Astra 40 steps and the other models 20. Do the differences among the leading models hold at a common budget, or do they reflect the exploration each model is allowed?

Response. They hold at a common budget. We ran both comparisons you suggest: the revised Table 1 runs every model at 20 steps, and the new App. F.9 (Table F7, Figure F1) gives pass rates at 20, 30 and 40 steps. At 20 steps GPT-6 Astra scores 27.5% and Claude Opus 5 25.2%. Pass rate (%) by step budget, 600 rounds, default judge:

Model20 steps30 steps40 stepsRounds that hit the 20-step cap
GPT-6 Astra27.5–30.54.5%
Claude Opus 525.225.727.00.5%
Claude Opus 4.822.8–25.06.2%
Claude Sonnet 520.820.721.33.3%
Gemini 3.8 Flash12.119.027.966.8%
Gemini 3.7 Flash18.726.027.241.0%
Gemini 3.6 Flash18.523.525.024.7%
Gemini 3.5 Flash3.010.013.089.2%

(Two rows differ slightly from the submission. The submitted GPT-6 Astra run at 40 steps (28.7%) was accessed through OAuth; here both budgets are run directly through the Vertex API. Gemini 3.7 Flash is the run graded by the default judge, like every other row; the submitted 20.3% was graded by Gemini 3.7 Flash.)

A step limit is part of any real deployment. 20 steps is about twice the mean length of the reference solutions, so the budget covers what the task needs. The leading models rarely reach it (0.5–6.2% of rounds) and gain at most 3 pp from twice the budget. So the differences among them reflect how they handle the task, not how much exploration they are allowed. The curve also shows how the models work. The Claude models are step-efficient: they finish in about as many steps as the reference solutions. The Gemini Flash models proceed in many small steps, often use the whole budget, and gain 6.5–15.8 pp from more steps. This is a difference in working style under the same budget. At 40 steps Astra still leads with 30.5%, and no model passes more, so the low pass rates do not come from the budget. Sec. 5.1 now compares all models at 20 steps, where the leading three models are the same as in the submission.

AuthorsOpenReview ↗

Response to Reviewer iuQe (2/2)

W2 and Q2. How should the co-failure result in Sec. 5.3 be described?

Response. Thank you for the suggestion; we adopt your wording. Sec. 5.3 and the Table 2 caption now describe two patterns: failures cluster on easy rounds (lift above 1) and are close to independent on hard rounds (lift near 1). The result itself is unchanged, and the second pattern explains our main finding. On hard rounds a model misses different requirements independently, so it completes most of a round but rarely all of it (GPT-6 Astra: mean check score 65.4%, 27.5% of rounds passed). In real use there is no reference solution and no checker to catch the missed step, which is why we rank models by the pass rate over all required checks. The per-check scores serve the analysis: they show which requirements each model misses, and so where it can be improved. Co-failure lift by round difficulty (revised Table 2, 600 rounds, 20 steps):

ModelEasy roundsMiddle roundsHard rounds
GPT-6 Astra1.481.051.05
Claude Opus 52.05*1.38*1.04
GPT-5.6 Sol1.121.001.03
Claude Opus 4.81.98*1.041.01
Kimi K32.21*1.17*1.05*
Claude Sonnet 51.72*1.17*1.04*
Gemini 3.7 Flash1.37*1.07*1.00
Gemini 3.6 Flash1.35*1.040.99

*The seed-clustered 95% interval excludes 1. Over all 15 models, lift is significantly above 1 on easy rounds for 11 models and ranges from 0.99 to 1.05 on hard rounds.

Q3. How were AI tools used in presenting the results, including the tables?

Response. All results are computed by our analysis scripts from the evaluation logs. As the Reproducibility Statement says, these scripts are released. We used an LLM to polish the script output tables as LaTeX tables and to polish the writing. The AI Use Statement have already disclosed the writing assistance and table polishing.

Reviewer_iuQeOpenReview ↗

Response to rebuttal

The authors have spent impressive amount of work in the rebuttal process. My main concerns are addressed.

With that said, I strongly recommend the authors to proofread and polish the figures, tables and text, to reduce the negative impression of AI-generated text.

I have raised my score to 3. Thank you.

Thank you so much for your positive response and helpful suggestions! We will polish the figures, tables and text of the revision paper.

Reviewer_3ZohOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces CLASHarness, a controllable generation framework and benchmark for evaluating how vision-language agents handle conflicting information from peer agents during computer use, for instance when a sub-agent reports a deployment as healthy while the dashboard shows failing health checks. Scenario synthesis is decomposed into an eight-stage, three-tier pipeline: (i) tier 1 sets up a consistent multi-agent workspace from structured seeds and plugins, (ii) tier 2 plans phases of 3-4 rounds and synthesises each round through eight sub-steps, including pre-committing the decisive ground-truth value, executing a reference trajectory in a sandbox and compiling checks from its recorded effects, and (iii) tier 3 audits consistency, renders the visual assets and scans for text leakage so that the ground-truth value appears only in images for visually grounded rounds.

The resulting benchmark contains 600 rounds across 20 seeds, 7,970 rendered visual assets and 3,284 scored checks. The authors evaluate 15 models and report four main findings:

  • weaker models trust misleading peer messages, while stronger models check the visual evidence,
  • fixing a conflict is harder than investigating it (-18.2 pp in pass rate),
  • closing actions, such as releasing locks, fail in over 58% of cases, and
  • more reasoning effort can trap weaker models in repetitive tool loops.
Critical strengths
  • The study is a timely and broad diagnostic of agent behaviour under conflicting peer information, a setting that existing computer-use benchmarks largely ignore.
    • It covers 15 recent models across different settings, such as failure modes, resolving vs investigating conflicts and inference cost.
  • The generation pipeline addresses well-known failure modes of LLM-generated agent benchmarks in a principled way. Committing the single ground-truth fact that settles the conflict in a round before rendering, scanning for its leakage into text and compiling checks from reference trajectories reduce answer leakage and unreachable rubrics.
    • The blind ablation supports the multimodal design: for the three models evaluated without images, pass rates fall to 11-17% of their sighted values.
Critical weaknesses
  • The "solvable by construction" guarantee is not fully supported. The paper claims that every task is solvable and that unreachable checks are eliminated. However,
    • Appendix G.11 reports 18 exact-match checks whose reference strings are not readable by the agent and pass only 0.7% of the time, and notification checks that depend on the message body rather than the recipient.
    • In several rounds, most models (10-14 of 14) message the right person, yet only 1-4 receive credit (Table G21). Since a round passes only if all required checks pass, such artifacts may directly lower pass rates.
    • I believe the paper would benefit from a human auditing a sample of rounds (task clarity, solvability and grading correctness) before the benchmark can serve as a reference.
  • Judging and run conditions are not always consistent across models. 65.1% of scored checks are LLM-judged without human validation. The default judge, Gemini 3.6 Flash, is also the default generator and an evaluated model, and the reported Gemini 3.7 Flash run was graded by Gemini 3.7 Flash itself, with no note in Table 1.
    • Pass-rate rankings are also less stable across judges than the main text suggests: under the Claude Opus 5 judge, ρ=0.949\rho = 0.949 and the leader changes (see Table F4).
    • GPT-6 Astra, ranked first in Table 1 but its advantage over Claude Opus 5 is not significant after correction (on lines 1196-1197). The three text-only models receive OCR transcripts of the images, which bypasses the exclusivity gate.
  • The multi-agent framing and the resolution gate are weaker than presented. The agent interacts with pre-recorded peer messages and logs rather than live peers (as also acknowledged in Limitations), and there is no conflict-free baseline. The resolve-vs-investigate gate also compares different rounds and seeds rather than "identical conflicts", and as noted in the Appendix only 41.7% of gated rounds contain the intended effects. With p=.052p = .052, I believe that the abstract's claim that resolving conflicts "drops task success significantly" seems too strong.
Questions for authors
  • Could you report a human audit on a random sample of rounds, including agreement between the semantic judge and human graders?
  • Do the semantic criteria accept functionally equivalent solutions that differ from the reference trajectory, such as alternative file paths or message wordings?
  • Do the task instructions explicitly state the closing actions checked by F5 (e.g. releasing a lock)? If not, part of the F5 failures may reflect under-specified instructions; an ablation with an explicit exit checklist would separate the two.
  • The main text states that GPT-6 Astra fails over 58% of closing actions, but Figure D1 covers only the ten 20-step models. Where does this figure for GPT-6 Astra come from?
  • Could you report GPT-6 Astra at 20 steps and Gemini 3.7 Flash graded by the default judge?
  • The parallel benchmark generated with Claude Sonnet 5 has a per-task trajectory-length correlation of only r=0.186r = 0.186 with the default one (noted on line 262). What does this imply for reproducing the benchmark with a different generator?
  • I believe the default semantic judge is not named in the main body of the paper.
10 comments
AuthorsOpenReview ↗

Response to Reviewer 3Zoh (1/8):

Thank you for the detailed and constructive review. We are glad that you find the study a timely and broad diagnostic of agent behavior under conflicting peer information, a setting that existing computer-use benchmarks leave open. We also appreciate that you find the generation pipeline principled (committing the deciding fact before rendering, leak scanning, checks compiled from reference trajectories) and the multimodal design supported by the blind ablation. Your suggestions led to several new experiments, and they strengthen these points.

We follow the order of your review: comments W1–W3 (with their sub-points a, b, …), then questions Q1–Q7. Where a question overlaps with a comment, we answer both together.

W1a. Is the "solvable by construction" guarantee supported, given the 18 exact-match checks in Appendix G.11?

Response. Yes. Solvability is the basis of the benchmark, and it holds for all 600 rounds. Every round has a reference trajectory, and the round's checks are compiled from its execution. When the corpus was built, each reference trajectory was executed in the environment and passed all of its round's checks. This is what "solvable by construction" means, and Sec. 3.1 now states it in these words.

Three results support this:

EvidenceResult
Passing episodes in the Table 1 runs1,832; none repeats the reference command sequence
Exit checklist that names the required state changes (264 rounds, three models)Rounds passed by at least one model rise from 72 to 147
Human audit: task is solvable from what the agent sees58 of 60 sampled rounds (96.7%)

The checklist result shows where the failures come from: the rounds can be solved, and models miss the closing actions they have to infer.

The 18 checks you point to are 0.5% of the 3,284 scored checks and lie in 18 rounds. The results do not depend on them: without those rounds, each model's pass rate moves by at most 0.9 pp and the ranking stays the same. The revised App. G.11 reports this.

W1b. In Table G21, most models message the right person, yet only 1–4 receive credit. Do these checks grade correctly?

Response. These checks grade what the message says as well as who receives it. A notification helps the recipient only if it states the facts the recipient needs, so each criterion lists those facts. The judge reads every message together with its recipient and channel, in the form "To ‹recipient› in #‹channel›: ‹message›".

In the rows with low credit, most models message the right person but leave out a required fact. One example is the lock round. The message to the inventory controller must state three facts: the lock status was verified as UNLOCKED, an exclusive lock was acquired, and the Order-7719 reservation is preserved. Twelve models message the controller. Four state all three facts, and these four receive the credit. Kimi K3, for example, reports the acquired lock and the preserved order but not the verified lock status, and receives no credit.

Round (recipient)The message must stateModels that message the recipientWhat most messages leave outModels with credit
Payment chargeback (support specialist)the verification code PASSPORT-VERIFY-9982 for flight LH-24011411 do not give the code3
Warehouse safety halt (safety officer)the 88.5 °C SCADA alarm conflicts with the 42.0 °C infrared thermometer reading, so the alarm is false1210 do not report the thermometer reading; 1 reports it in a different channel1
Distributed lock (inventory controller)lock verified UNLOCKED, exclusive lock acquired, Order-7719 preserved128 leave out at least one of the three facts4
Expiring resource hold (research lead)hold token HOLD-80G-9412-PROVISIONAL acquired with a 300 s TTL106 do not give this token (3 give a different one); 2 announce the hold before placing it2

Re-running the default judge on the same messages gives the same credit in all four rows. Six additional models from three vendors also re-graded 26 judge verdicts on the Table G21 checks with the evaluated model hidden, and their majority agrees with the judge on 23 of them. The revised App. G.11 states that the judge sees the recipient and the channel.

AuthorsOpenReview ↗

Response to Reviewer 3Zoh (2/8)

W1c and Q1. A human audit of sampled rounds should confirm task clarity, solvability and grading correctness, including agreement between the semantic judge and human graders.

Response. Thank you for this suggestion. The revision includes this audit: a summary in Table 4a (Sec. 6) and the full results in App. F.8 (Table F6). Three annotators independently audit 60 rounds (three per seed, one per difficulty level) and the 337 required checks of these rounds. They work from what the agent can see. Each result is the majority answer of the three annotators.

Audit questionItemsResult (majority of three)Agreement among annotators
Instruction is clear60 rounds98.3% (59/60)92.8%
Task is solvable60 rounds96.7% (58/60)92.2%
Correct decision is unique60 rounds95.0% (57/60)87.8%
Reference trajectory is correct60 rounds96.7% (58/60)90.0%
Check is reasonable337 checks96.1% (324/337)91.2%
Instruction names or implies the check337 checks92.9% (313/337)87.1%
A reasonable alternative would fail the check337 checks5.0% (17/337)90.3%
Annotators agree with the judge verdict170 verdicts94.7% (161/170), κ = 0.8992.9%

The annotators find the sampled rounds clear and solvable and the checks reasonable in the large majority of cases.

Agreement with the semantic judge. Six additional models from three vendors re-graded 170 judge verdicts from the same 60 rounds with the evaluated model hidden. Their majority agrees with the judge on 98.2% of the verdicts (κ = 0.96; three ties not counted). The annotators grade the same verdicts (last row of the table above; Table F6). The six-model review is in App. F.7 (Table F5).

AuthorsOpenReview ↗

Response to Reviewer 3Zoh (3/8)

W2a. How reliable are the grades of the LLM judge, which grades 65.1% of the scored checks?

Response. The grades are reliable. Three independent tests support this.

  1. Repeated votes. The judge's three votes agree on 97.5% of checks. Even if every split vote is flipped, a model's pass rate moves by at most 2.3 pp and its rank by at most one place.
  2. Full re-grades by other judges. We re-graded all semantic checks of 14 models with judges from the other two vendors, including the newer GPT-6.1 Sol, and re-ran the default judge. Agreement with the recorded grades:
Re-grading judgeκ on checksRank correlation, pass rateRank correlation, mean score
Claude Sonnet 50.9220.9910.991
Claude Opus 50.9020.9490.987
GPT-6.1 Sol (new)0.9210.9530.987
Gemini 3.6 Flash (re-run)0.9770.9750.987
  1. Review by six additional models. Six additional models from three vendors each re-graded a sample of 170 judge verdicts with the evaluated model hidden. Agreement with the judge:
Reviewing modelAgreement with the judgeκ
GPT-6 Astra95.3%0.90
GPT-6.1 Sol95.9%0.91
Claude Opus 5.597.1%0.94
Claude Fable 5.196.4%0.92
Gemini 4 Argon95.9%0.91
Gemini 4 Carbon93.5%0.87
Majority of the six (three ties not counted)98.2%0.96

The annotators of the human audit grade the same 170 verdicts, and App. F.8 (Table F6) reports their agreement with the judge. The re-grades are in App. F.5 (Table F4), and the six-model review is in App. F.7 (Table F5).

The main diagnostic findings (visual grounding, evidence citation, closing actions, deadlines) use deterministic checks only, so they are independent of the judge.

W2b. The default judge, Gemini 3.6 Flash, is also the generator and an evaluated model. Does this favor Gemini models?

Response. The default judge treats Gemini models like every other model. Two results show this.

When a judge from another vendor takes over, the scores of Gemini models change as much as those of the other models. Mean change in pass rate relative to the default judge (pp, 14 models re-graded in full):

JudgeGemini (4 models)Claude (3)OpenAI (3)
Claude Sonnet 5+0.4−0.8−0.2
Claude Opus 5+2.1+1.5+3.0
GPT-6.1 Sol (new)−0.6−2.1−0.7

The six reviewing models agree with the default judge as often on Gemini outputs as on other models' outputs (170 sampled verdicts, majority of the six, three ties not counted):

Evaluated modelVerdictsAgreement with the default judge
Gemini 3.6 Flash5398.1%
GPT-5.6 Sol62100.0%
Claude Opus 55296.2%

A corpus generated by Claude Sonnet 5 also reproduces the model ranking (ρ = 0.96).

W2c and Q5. What do Gemini 3.7 Flash and GPT-6 Astra score under the default judge and at 20 steps?

Response. Both are in the revised Table 1, which now uses one judge and one step budget for every model. Pass rate (%), 600 rounds, 20 steps, default judge:

RunPass rate
GPT-6 Astra, 20 steps27.5
Gemini 3.7 Flash, High effort18.7
Gemini 3.7 Flash, Medium effort19.3
Gemini 3.7 Flash, Low effort17.8

Astra stays in the leading group. For Gemini 3.7 Flash, this run and the submitted High run agree when graded by the same judge (p = .63).

W2d. Under the Claude Opus 5 judge, the pass-rate ranking has ρ = 0.949 and the leader changes. Is the ranking stable across judges?

Response. Yes. Rankings agree closely under every judge, including the new GPT-6.1 Sol judge. Rank correlation with the default judge (14 models):

Re-grading judgePass rateMean score
Claude Sonnet 50.9910.991
Claude Opus 50.9490.987
GPT-6.1 Sol (new)0.9530.987
Gemini 3.6 Flash (re-run)0.9750.987

The change of leader is a swap between Claude Opus 5 and GPT-5.6 Sol. Under the default judge these two models are 0.7 pp apart, well within one standard error (2.5–3.0 pp). The same four of the 14 re-graded models rank highest under every judge. The revised Sec. 4 reports both metrics, and Table F4 (App. F.5) gives them for each judge.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all

We thank all three reviewers for their careful and constructive reviews. We are encouraged by the strengths they see in the paper:

  • A timely setting. Agents that act on conflicting information from their peers are a setting that current computer-use benchmarks leave open, and the paper gives a broad diagnostic of it across 15 recent models (3Zoh, eGsC).
  • A principled pipeline. Committing the deciding fact before rendering, scanning for leaks and compiling checks from executed trajectories keeps instruction, evidence and grading consistent, and is useful well beyond this benchmark (all reviewers).
  • Genuinely multimodal tasks. The tasks need the information in the images and offer no text-only shortcut: without images, pass rates fall to 11–17% of their sighted values, also on tasks whose checks do not ask the agent to open an image (3Zoh, iuQe).
  • Practical findings. The gap between per-check scores and full pass rates, and the consistent failure to complete closing actions such as releasing locks, are findings that practitioners can use (eGsC).

The reviews share five main questions. We answer them here; other points are answered in our reply to each reviewer.

Question 1. Why are the pass rates low?

The tasks are reasonable. Three annotators audit 60 rounds and their 337 checks. They find 96.7% of the rounds solvable and 96.1% of the checks reasonable. Every round has a reference solution that passed all of its checks at build time. Six additional models from three vendors re-grade 170 judge verdicts and agree with 98.2%.

The failures follow one pattern. Frontier models decide correctly and then leave the resolution unfinished. Their reports state the correct decision in 84–85% of cases. What they miss are closing actions, such as releasing a lock, that the instruction leaves to the evidence. Once a checklist names these actions, the models perform 93–100% of them. The benchmark therefore shows a concrete gap: when a conflict that nobody planned for comes up, current models do not work out by themselves what still has to be done.

ModelReport states the correct decision (%)Closing actions performed, without → with checklist (%)
Claude Opus 585.036.3 → 100.0
GPT-6.1 Sol84.228.4 → 95.1
Gemini 3.6 Flash84.227.5 → 93.1

Question 2. Is the step budget the same for all models?

Yes. The revised Table 1 runs every model at 20 steps, about twice the length of the reference solutions. GPT-6 Astra scores 27.5% and stays in the leading group. Twice the budget adds at most 3 pp for the leading models. The Gemini Flash models gain more because they work in many small steps. At both budgets the best pass rate stays near 30%.

Model20 steps (%)40 steps (%)Rounds that hit the 20-step cap
GPT-6 Astra27.530.54.5%
Claude Opus 525.227.00.5%
Claude Opus 4.822.825.06.2%
Gemini 3.7 Flash18.727.241.0%
Gemini 3.8 Flash12.127.966.8%

Question 3. Are 20 seeds enough to support the conclusions?

Yes. The revised tables report uncertainty over seeds. A seed is a general scenario description. The generator samples three packages of ten rounds from each seed at temperature 0.4, and the rounds differ widely: 2,086 of the benchmark's 2,137 semantic criteria are distinct. The leading models are peers. GPT-6 Astra, Claude Opus 5 and GPT-5.6 Sol are frontier models of the same generation, so close scores among them are expected. The main finding is clear. Models reach high partial scores but do not finish the task: the best one passes 65% of the checks and 27.5% of the rounds.

Question 4. Does the wording of the instruction change the results?

Yes, and the change supports the design. Our instructions describe the competing claims without saying which one is correct. When the conflict is not described, the agent has to notice it by itself, and pass rates fall. When the correct decision is given, pass rates rise but reach at most 31%. Knowing who is right is not enough: the agent still has to carry the resolution through.

ModelConflict not described (%)Original instruction (%)Correct decision given (%)
Claude Opus 514.220.830.8
GPT-6.1 Sol16.723.326.7
Gemini 3.6 Flash16.718.320.0

Question 5. Why are the peers agent scripted?

By design, for control and reproducibility. Every model faces exactly the same conflict and the same peer behavior, and every result can be reproduced. For the evaluated agent, the setting is still multi-agent: it messages the peer agents, delegates tasks to them and receives their replies and results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.