acceptodds
Under review as a conference paper at ICLR 2027

GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution

Abstract

The executable harness surrounding a GUI model determines how observations are assembled, actions are executed, and verification, recovery, and termination are controlled. Compared with harness optimization for non-GUI agents, automatically optimizing this harness poses three coupled challenges: reconciling model intent with observed visual effects, diagnosing failures under variable execution outcomes, and identifying recurrent failure patterns across tasks and translating them into reusable runtime changes. We introduce GUI-HARVEST, an automatic harness optimizer that enables self-improving GUI agents with frozen backbone models. First, to ground diagnosis in observed action effects, it aligns model outputs and executed actions with before-and-after screenshots, producing findings tied to specific interface transitions. Second, to account for execution variability, it treats repeated executions of the same task as a joint evidence unit, using within-task comparisons to locate outcome-relevant behavioral differences. Third, to derive reusable interventions from task-local evidence, it consolidates verified findings across tasks into recurring failure patterns, maps them to bounded source-code edits with predictions recorded before evaluation, and checks the predicted behavioral effects alongside task performance through repeated execution. Experiments on OSWorld-Verified show consistent held-out gains across six general-purpose open, GUI-specialized open, and proprietary backbone models; Qwen3-VL-32B-Instruct gains 12.33 points on the full suite. Frozen-harness transfer improves GPT-5 by 13.87 percentage points on WindowsAgentArena at 50 steps without further optimization. With the same backbone and initial harness, GUI-HARVEST outperforms Self-Harness and Meta-Harness, suggesting that GUI-specific diagnosis and validation help produce harness improvements that generalize to unseen tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI agents, agent harness, self-improving agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_C5L7OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GUI-HARVEST, a method for improving GUI agents by editing their execution harness while keeping the backbone model fixed. It analyzes multiple multimodal execution traces, clusters failure patterns across tasks, proposes targeted harness modifications, and validates candidate modifications using both measured utility and a soft check of whether predicted behavioral changes actually occurred. The experiments use an 80/80/201 Search/Validation/Test split of OSWorld-Verified, six backbone models, comparisons with other harness optimization methods, ablations, and transfer tests on WindowsAgentArena. On the held-out 201-task OSWorld-Verified Test split, Qwen3-VL-32B improves from 43.02% to 53.18% (+10.16 percentage points), with improvements also reported for the other evaluated backbones.

Critical strengths
  1. Meaningful evaluation beyond the harness-development split. The method is tested on a separate 201-task set with six different backbones, which helps distinguish general improvements from merely memorizing the tasks used to edit the harness. The reported +10.16 percentage-point result for Qwen3-VL-32B is a substantial effect on this held-out set.

  2. A concrete, analyzable optimization mechanism. Instead of treating failed GUI executions as undifferentiated feedback, the method combines visual/action traces, repeated trials, shared failure patterns, and targeted harness patches. Ablations and WindowsAgentArena transfer results provide useful, although not definitive, support for this design.

Critical weaknesses
  1. The cost evidence does not establish end-to-end cost efficiency. The reported inference/API cost savings concern evaluation with a frozen, optimized harness. Appendix G.10 explicitly excludes optimization-process costs. The search and validation loops, repeated environment executions, and model-based analysis may be expensive enough to change the practical cost comparison substantially. This matters because lower operating cost is an important part of the paper's appeal.

  2. The behavioral-gating mechanism and optimization stability are not fully isolated. The soft gate relies on a model's judgment about predicted behavioral changes, yet its semantic accuracy is not independently established. The ablations do not fully equalize optimization trajectories and budgets, and a single search/validation/test split does not characterize variability across independent optimization runs. Consequently, the data support the overall pipeline more strongly than the precise causal contribution of the gate.

Questions for authors
  1. What is the complete cost of one optimization run, including model tokens/API charges, environment or VM hours, and repeated rollouts? How many subsequent evaluation or deployment tasks are required to amortize this cost?

  2. On a human-audited sample of proposed harness patches, how often does the soft gate correctly accept or reject the predicted behavioral change? Does it outperform a score-only gate at a matched proposal and execution budget?

  3. Can the authors report a repeated optimization run or a different split for at least one backbone, together with task-paired uncertainty on the final held-out Test improvement?

3 comments
AuthorsOpenReview ↗

Response to Reviewer C5L7 (Part 1/2)

Thank you for your careful review and concrete suggestions. We appreciate your recognition of our held-out evaluation and the inspectable optimization mechanism.


Response to W1 and Q1

Q1. What is the complete cost of one optimization run, including model tokens/API charges, environment or VM hours, and repeated rollouts? How many subsequent evaluation or deployment tasks are required to amortize this cost?

Optimization costs. We agree that the frozen-harness inference costs in Section 4.4 and Appendix G.10 do not include optimization costs. Following your suggestion, we compiled the costs of the original campaigns from execution and usage logs. Each evaluated harness uses 80 Search tasks, 80 Validation tasks, three executions per task, and a 15-step limit. Rollout totals include the initial harness and all evaluated candidates.

Target backboneEvaluated candidatesTotal rolloutsTarget-model tokens (input / output)Target-model API (USD)Optimizer tokens (input / output)Optimizer API (USD)Cumulative VM hoursVM cost (USD)Rollout wall-clock time (h)Total cost (USD)
Gemini 3.1 Pro189,1201.11B / 53M2,67251.7M / 3.3M711,29565222,808
GPT-5168,1601.26B / 25M1,67346.6M / 3.4M701,91095321,838
Qwen3-VL-32B2512,4804.94B / 73MSelf-hosted77.9M / 6.1M1263,73018662312 + GPU
Qwen3-VL-8B168,1602.86B / 27MSelf-hosted50.8M / 4.7M921,3586823160 + GPU
OpenCUA-72B178,6400.75B / 10MSelf-hosted48.7M / 3.9M791,0735418133 + GPU
OpenCUA-32B147,2000.67B / 34MSelf-hosted40.3M / 3.4M691,0855418123 + GPU

B and M denote billion and million tokens. Optimizer charges include all four Claude Sonnet 5 roles and prompt caching. VM hours are cumulative, while wall-clock time covers rollouts only. For GPT-5 and Gemini 3.1 Pro, optimizer API charges account for only 3% to 4% of the total cost of an optimization run.

Amortization. At the operating points reported in Section 4.4, our 15-step GPT-5 harness achieves 62.4% Full accuracy, compared with 62.6% for published Agent S3 at 100 steps, at approximately USD 0.20 vs. USD 0.72 in API cost per task. At similar accuracy, the USD 0.52 per-task API saving amortizes the measured USD 1,838 optimization cost after approximately 3,500 task executions. Counting deployment API calls and this one-time cost, savings reach 47% after 10,000 tasks and 67% after 50,000 tasks relative to that Agent S3 operating point, assuming comparable future workloads and the reported prices. In this case, reusing the GPT-5 harness can recover the upfront optimization cost and yield increasing net savings at similar accuracy.

Cost per successful task. At the 15-step operating point in Figure 9, the frozen harness optimized by GUI-HARVEST reduces API cost per successful task by approximately 19% for GPT-5, 40% for Gemini 3.1 Pro, and 5% for Qwen3-VL-32B, relative to the initial harness. For Qwen3-VL-32B, the full-suite cost rises from USD 27.3 to USD 34.2, but accuracy increases from 38.6% to 50.9%, explaining the lower cost per success. These findings show that the optimized harnesses reduce the API cost per successful task during deployment.

▼▼ Continued in Part 2/2. ▼▼

AuthorsOpenReview ↗

Response to Reviewer C5L7 (Part 2/2)

▲▲ Continued from Part 1/2. ▲▲

Response to W2 and Q2

Q2. On a human-audited sample of proposed harness patches, how often does the soft gate correctly accept or reject the predicted behavioral change? Does it outperform a score-only gate at a matched proposal and execution budget?

Human audit of the soft gate’s accept/reject decisions. We sampled 18 candidate patches across all 6 original backbones, including all 35 required predictions and 111 prediction–task judgments for those patches. An annotator reviewed the frozen predictions, pre-edit findings, and post-edit executions, with model verdicts, patches, and scores withheld. Task-level agreement with Sonnet 5 was 98/111 (88.3%). For the 18 audited harness edits, applying the original aggregation rule to the human task-level verdicts reproduced the behavioral gate’s pass/reject decisions for 17/18 edits (94.4%). The breakdown is shown below.

Original behavioral-gate decisionReviewed candidatesConfirmed by human-label aggregation
Pass1110
Reject77
Total1817 (94.4%)

The behavioral-gate disagreement was in a candidate already rejected by the score gate. Therefore, replacing the model verdicts with human verdicts left the final accept/reject decisions unchanged for all 18 audited harness patches.

Full-campaign cross-model verification. To check whether changing the validator model would alter the recorded promote/rollback decisions, we used GPT 6.1 Sol and Kimi K3 to separately reassess all 770 judgments from 231 predictions across 106 candidates. Their agreement with Sonnet 5 was 92.3% and 89.1%, respectively. Replaying each model's labels under the original aggregation and score gates preserved all recorded promote/rollback decisions. Task-level verification tables appear in our response to Reviewer KLkj (Part 3/3, W3 and Q3).

Comparison at matched budgets. We extended the score-only run from 21 to 25 evaluated candidates. All four additional candidates were rejected, leaving its selected harness unchanged. Both variants use the same Search/Validation score gate and 12,480 rollouts, including the initial harness.

Qwen3-VL-32B resultScore-only gateScore + behavioral gates
Evaluated candidates2525
Test (%)49.3653.18 / 52.62 (two runs)

Both full-method runs outperform score-only. Their mean advantage over the score-only run is 3.54 points, with a task-paired 95% confidence interval of 1.3 to 5.8 points. These results show that, for Qwen3-VL-32B, adding the behavioral soft gate improves held-out Test performance over score-only gating at matched proposal and execution budgets.


Response to W2 and Q3

Q3. Can the authors report a repeated optimization run or a different split for at least one backbone, together with task-paired uncertainty on the final held-out Test improvement?

Independent repeated optimization run. Following your suggestion, we added an independent repeated optimization run using Qwen3-VL-32B, with the same data split, initial harness, and budget, changing only the optimizer roles' sampling seeds.

Optimization runTest (%)Full (%)Test gain over initial harness (pp)
Original53.1850.94+10.16
Additional repeated run52.6250.47+9.60

The repeated run's Test score is only 0.56 points below the original and remains above all reported baselines. Both runs independently discovered the same four types of edits: commit checks, routing between GUI and code execution, retirement of ineffective routes, and screen-difference evidence. The similar held-out gains and recurring mechanisms support repeatability.

Task-paired uncertainty. We jointly resampled the 201 Test tasks across the initial and optimized harnesses, keeping each task's three executions together. Differences and 95% confidence intervals below are in percentage points.

BackboneTest improvement95% confidence interval
Qwen3-VL-32B+10.165.8 to 14.5
Qwen3-VL-8B+6.052.2 to 9.9
Gemini 3.1 Pro+6.202.6 to 9.8
GPT-5+5.511.9 to 9.1
OpenCUA-72B+2.930.8 to 5.1
OpenCUA-32B+1.49-0.5 to 3.5

All six backbones show positive observed improvements. The intervals quantify evaluation uncertainty for the selected harnesses, with five lying entirely above zero. OpenCUA-32B's modest gain is consistent with the limited adaptation flexibility of its native action interface discussed in Section 4.3. We will add the paired estimates and independent restart results to the revision.


Thank you again for your constructive feedback. We hope the cost accounting, gate audits, matched-budget comparison, and independent restart address your concerns.

AuthorsOpenReview ↗

Gentle Reminder: Our Rebuttal and Revised Manuscript

Dear Reviewer C5L7,

Thank you again for your careful review. We would appreciate your consideration of Parts 1/2 and 2/2 of our response, which provide the following additional evidence:

  • Optimization costs and amortization: We report token/API usage, rollout counts, and VM resources. The GPT-5 campaign costs USD 1,838. At the reported operating points, API savings over Agent S3 recover this cost after approximately 3,500 tasks, assuming comparable workloads and prices.
  • Human and cross-model audits: Human verdicts reproduce 17/18 behavioral-gate decisions (94.4%), while all 18 final promote/rollback decisions remain unchanged. Alternative-validator replays also preserve all 106 recorded final decisions.
  • Matched-budget gate comparison: With 25 candidates and 12,480 rollouts per campaign, the two full-method runs average 3.54 Test points above score-only gating on Qwen3-VL-32B, with a task-paired 95% CI of [1.3, 5.8].
  • Optimization repeatability and uncertainty: An independent Qwen3-VL-32B restart reaches 52.62% Test versus 53.18% originally and recovers similar edit mechanisms. We also provide task-paired confidence intervals for all six original backbone gains, with five lying entirely above zero.

These additions are included in the revised manuscript. We welcome any remaining questions and would be happy to clarify them before the discussion closes. Thank you for your time and consideration.

Reviewer_NjzWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GUI-HARVEST, a framework for automatically improving GUI agents by modifying their runtime harnesses. It compares repeated executions of individual tasks using action logs and screenshots, groups observed failures across tasks, and generates code changes targeting shared weaknesses. Candidate changes are accepted when they satisfy search and validation score criteria and receive evidence supporting predefined behavioral predictions. Experiments on OSWorld-Verified report held-out improvements across six backbone models and higher scores than the evaluated harness optimization baselines. Harnesses optimized on OSWorld also improve two models on WindowsAgentArena without further optimization.

Critical strengths
  1. Addresses an important practical problem. GUI-agent reliability depends on runtime decisions about observation construction, tool execution, recovery, and termination. Adapting these mechanisms to different models requires manual engineering. The paper studies how to automate this process using execution feedback while keeping model weights fixed, offering a practical direction for improving existing agents.
  2. Encouraging results across multiple backbones and environments. The selected harnesses improve reported OSWorld-Verified Test scores across six backbones spanning general-purpose open models, GUI-specialized models, and proprietary models. Qwen3-VL-32B gains 10.16 percentage points, while GPT-5 and Gemini 3.1 Pro also improve. Transferring the frozen harnesses to WindowsAgentArena yields additional gains, including 13.87 points for GPT-5, without further harness optimization. These experiments provide evidence of usefulness beyond a single model or benchmark.
  3. A well-motivated design with inspectable evidence-to-edit examples. The framework combines repeated task executions, screenshots of action effects, cross-task failure aggregation, and behavioral predictions recorded before evaluation. These choices address characteristic GUI difficulties, including variable execution outcomes and premature completion. The appendix traces a concrete intervention from contrasting execution paths through diagnosis, code modification, and post-edit behavior checks. The optimization traces and failure-pattern summaries provide further visibility into the changes produced by the system.
Critical weaknesses
  1. The contribution needs clearer differentiation from closely related work. The paper covers a broad literature, but several closely related methods, including HarnessLens and HarnessEvolve, receive only brief discussion. These methods also connect execution evidence to harness updates and use additional checks to guide acceptance. The paper should explain more precisely how its diagnostic evidence, cross-task aggregation, and behavioral validation differ from these approaches. This distinction matters because the performance of the complete pipeline alone does not establish which methodological advances account for its contribution.

  2. The evidence for reliable cumulative self-improvement remains incomplete. The optimization traces, failure-pattern counts, and end-to-end example help explain how the system operates. However, candidates are accepted only when Search and Validation scores do not decline. Rising curves on these splits therefore partly reflect the selection rule and cannot independently demonstrate that successive edits reliably correct the intended failures. The final held-out gains are encouraging, but the paper does not report whether independently repeating the optimization produces similarly effective harnesses. Repeating task execution three times measures a different source of variability. Patch-level contribution analysis would also help distinguish useful interventions from changes whose benefits depend on other edits. Finally, a behavioral prediction can pass with one supporting task verdict while the remaining verdicts are inconclusive, making coverage statistics and independent audits important for assessing the reliability of this check.

  3. The optimizer comparisons do not adequately control evaluation resources and selection feedback. GUI-HARVEST permits up to five evaluated candidates per round and uses both Search and Validation scores for acceptance. Meta-Harness proposes two candidates per round and selects using Search scores alone. Consequently, the shared ten-round cap does not give the methods equivalent opportunities to discover and select effective changes. The reported results support the evaluated GUI-HARVEST configuration, but leave uncertainty about how much of its advantage comes from the proposed mechanisms. A comparison with matched rollout budgets and selection rules would provide stronger evidence.

  4. The methodological conclusions are not sufficiently validated on stronger backbones. The reported Test improvements on GPT-5 and Gemini 3.1 Pro provide useful evidence that the complete framework benefits stronger models. However, comparisons against generic harness optimizers and all component ablations are restricted to Qwen3-VL-32B. These experiments therefore leave open whether the relative advantage and individual component benefits persist across models with different failure profiles. At least one controlled comparison or component analysis on a stronger backbone would strengthen the generality claim. The limitations discussion should also distinguish conclusions supported across backbones from those established only in the Qwen experiments.

Questions for authors
  1. What specific capabilities distinguish your diagnosis and validation mechanisms from the closest harness-evolution methods, particularly HarnessLens and HarnessEvolve?
  2. How reproducible are the gains across independent optimization runs? Can you provide patch-level contribution analysis and report behavioral-validation coverage and audited accuracy?
  3. Does the advantage over Meta-Harness persist with matched rollout budgets and the same Search/Validation selection protocol?
  4. Can you provide comparative or ablation evidence on a stronger backbone, and discuss which failure patterns or settings offer limited improvement?
4 comments
AuthorsOpenReview ↗

Response to Reviewer NjzW (Part 1/3)

Thank you for your thoughtful review and constructive suggestions. We appreciate your recognition of the practical importance of harness adaptation and our results across models and environments.


Response to W1 and Q1

Q1. What specific capabilities distinguish your diagnosis and validation mechanisms from the closest harness-evolution methods, particularly HarnessLens and HarnessEvolve?

We agree that these distinctions deserve clearer discussion. Both are concurrent works first posted to arXiv approximately three to four weeks before the submission deadline and are already cited in Appendix A. The following comparison identifies the specific differences.

AspectHarnessEvolveHarnessLensGUI-HARVEST
Diagnostic evidenceFailed trajectories compared with verified, answer-conditioned reference trajectories when available; otherwise analyzed independentlyReusable experiences and recurring deficiencies extracted from model-visible trajectories and outcomesRepeated executions of the same task, model, and harness, with actions aligned to before-and-after screenshots
Cross-task aggregationBatch-level error clustersRecurring behavioral experiencesFailure modes with explicit membership tests and behavioral targets, supported by at least two independent tasks
Update validationChecks for answer leakage and excessive insertion of in-context examples, plus performance gatesPaired incumbent/candidate trials, behavioral review, and confirmation trialsPredictions frozen before evaluation, assessed by a separate task-local Validator blind to the patch, edit rationale, and scores

Our GUI-specific diagnosis. GUI-HARVEST uses successful sibling runs, when available, as comparisons without supplying reference answers. Its Evidence Analyst examines screenshots and landing-point crops to distinguish an intended action from its actual visual effect. Its deterministic verifier checks cited steps and mechanical facts, while semantic interpretation remains model based.

Our behavioral validation. HarnessLens also hides the patch during its initial behavior comparison. Our distinction is that observable tests are frozen before evaluation and assessed through task-local, score-blind supported/contradicted/inconclusive verdicts. We will make this comparison explicit in the revision.

Component evidence. To address your concern about which methodological advances account for the gains, we examine the contribution of each distinguishing component. The first three ablations below are from Table 3 of the paper. To examine the benefit of jointly comparing repeated runs of the same task during failure diagnosis, we additionally evaluated independent per-run diagnosis followed by pooled findings, using the same evaluation budget and promotion rule.

Qwen3-VL-32B configurationTest (%)Change from full method (pp)
Without visual evidence44.54-8.64
Without Cross-task Clusterer45.66-7.52
Hard gate only49.36-3.82
Without joint comparison (independent per-run diagnosis, new)50.66-2.52
Full GUI-HARVEST53.180.00

These results support the contributions of visual evidence, cross-task aggregation, behavioral validation, and joint diagnosis within our pipeline. Direct OSWorld comparisons with HarnessLens and HarnessEvolve require additional GUI adapters and full optimization runs, which we could not complete within the discussion period.

▼▼ Continued in Part 2/3. ▼▼

AuthorsOpenReview ↗

Response to Reviewer NjzW (Part 2/3)

▲▲ Continued from Part 1/3. ▲▲

Response to W2 and Q2

Q2. How reproducible are the gains across independent optimization runs? Can you provide patch-level contribution analysis and report behavioral-validation coverage and audited accuracy?

We agree that rising Search/Validation curves partly reflect the acceptance rule. We therefore report an independent restart, cumulative and patch-level analyses on the sealed Test split after optimization stopped, behavioral-validation coverage, and human and cross-model audits.

Independent restart. Following your suggestion, we added an additional independent optimization run on Qwen3-VL-32B with the same setting, including the same data split, initial harness, and optimization budget, but different optimizer sampling seeds. It achieved a Test score of 52.62%, 0.56 percentage points below the original 53.18% and above the initial harness's 43.02%. Both runs independently discovered the same four types of harness edits: adding commit checks, routing between GUI and code execution, retiring ineffective execution routes, and incorporating screen-difference evidence. The similar Test scores and recurring edit mechanisms support the reproducibility of the observed gains.

Cumulative improvement and failure reduction. We additionally report Test scores for each intermediate harness (Qwen3-VL-32B), alongside the initial and final scores from Table 1 of the paper: 43.02, 46.12, 47.31, 48.85, 49.63, 50.47, 51.92, and 53.18. Thus, all seven accepted updates improved held-out performance. For three targeted failure modes, namely uncommitted edits at completion, repeated ineffective actions, and inefficient GUI routes, the numbers of affected Test tasks decreased by 61%, 55%, and 47%, respectively, compared with 7% for untargeted residual failures. These results suggest that the harness edits address their intended failure modes and generalize to unseen tasks. Together, these findings support cumulative improvement through successive corrections to the intended failures on Test tasks that did not guide patch acceptance.

Patch-level contribution analysis. Removing each accepted patch from the final harness reduced Test performance. Effects for four selected patches are shown below, with changes in percentage points.

PatchRemoved from final harnessAdded alone to initial harness
Commit gate and completion adjudicator-2.9+3.4
Screen-difference evidence-2.3+2.1
Code-agent protocol-1.6+0.9
Goal binder-0.7-0.3

The goal binder helps within the final harness despite hurting performance alone, illustrating that patch benefits can depend on other edits.

Behavioral-validation coverage statistics. Across 770 task-level validation judgments from six backbones, each assessing one behavioral prediction on one task, 61% were supported, 21% contradicted, and 18% inconclusive. The behavioral gate rejected 19 candidates that passed the score gate across six backbones. Among inconclusive judgments, 61% reflected absent triggers, 14% missing evidence, and 25% ambiguous behavior. Among the 66 predictions associated with accepted harness edits, only 5 had one supporting judgment and inconclusive judgments on all other tasks, where triggers were absent. 3 cases came from OpenCUA, whose action-normalization edits trigger on relatively few tasks. For the 2 Qwen3-VL-32B cases, individually removing the associated edits from the final harness reduced Test scores by 1.1 and 0.4 percentage points. Thus, this acceptance pattern is uncommon, and the corresponding examined edits still contribute to held-out performance.

Human and cross-model verification. We sampled 18 complete candidates from the 6 original backbones, covering all 35 predictions and 111 prediction–task judgments. An annotator assessed the frozen predictions against execution evidence, with model verdicts and scores withheld. Sonnet 5 matched the human labels on 98/111 judgments (88.3%). We reapplied the original aggregation rule using human verdicts in place of Sonnet 5’s. The behavioral gate’s pass/reject decisions were unchanged for 17/18 audited harness edits (94.4%). The remaining edit had already failed the score gate, so the final promote/rollback decisions were unchanged for all 18 edits. To check consistency across validators, we used GPT 6.1 Sol and Kimi K3 to separately reassess all 770 judgments. Agreement with Sonnet 5 was 92.3% and 89.1%, respectively. Replaying either model’s verdicts under the original gates preserved the final promote/rollback decisions for all 106 candidates. Detailed tables appear in our responses to Reviewers KLkj (Part 3/3, W3 and Q3) and C5L7 (Part 2/2, W2 and Q2).

▼▼ Continued in Part 3/3. ▼▼

AuthorsOpenReview ↗

Response to Reviewer NjzW (Part 3/3)

▲▲ Continued from Part 2/3. ▲▲

Response to W3 and Q3

Q3. Does the advantage over Meta-Harness persist with matched rollout budgets and the same Search/Validation selection protocol?

Following your suggestion, we added comparisons in which Meta-Harness and GUI-HARVEST use the same rollout budget and the same score feedback for selection: both use Search scores only. The advantage of GUI-HARVEST persists after matching rollout budgets and access to score feedback.

We extended Meta-Harness to 25 candidates (same as GUI-HARVEST) under its native proposal policy and evaluated GUI-HARVEST without Validation feedback (using the same score feedback as Meta-Harness). Both Search-only configurations use 6,240 rollouts, including the initial harness.

Qwen3-VL-32B configurationCandidatesScore feedbackTest (%)
Meta-Harness, original20Search only47.40
Meta-Harness, extended25Search only47.92
GUI-HARVEST without Validation25Search only (behavioral gate retained)50.68
Full GUI-HARVEST25Search + Validation53.18

The Search-only comparison retains a 2.76-point advantage for GUI-HARVEST, with a paired 95% CI of [+0.9, +5.3]. Adding Validation score feedback raises GUI-HARVEST's Test score from 50.68% to 53.18%, a gain of 2.50 percentage points. These results show that additional candidates or Validation feedback do not fully explain the advantage. The advantage over Meta-Harness therefore persists with matched rollout budgets and score feedback, providing stronger evidence for the proposed mechanisms. We will clarify the scope of budget matching in the revision.


Response to W4 and Q4

Q4. Can you provide comparative or ablation evidence on a stronger backbone, and discuss which failure patterns or settings offer limited improvement?

Following your suggestion, we evaluated Meta-Harness on GPT-5, sharing the initial harness and optimizer model. We report both its 16-candidate checkpoint and its full ten-round run. Each method retains its native selection policy.

GPT-5 configurationEvaluated candidatesTest (%)
Initial harness056.35
Meta-Harness, eight rounds1658.10
Meta-Harness, ten rounds2058.42
GUI-HARVEST1661.86

GUI-HARVEST improves over Meta-Harness by 3.76 points at equal candidate counts and 3.44 points even when Meta-Harness evaluates four additional candidates. The respective 95% CIs are [+1.4, +6.1] and [+1.1, +5.8], both excluding zero and hence supporting GUI-HARVEST's advantage on this stronger backbone.

Component analysis on GPT-5. In the optimization records, 71% of findings supporting accepted edits relied on decisive visual evidence, and 54% used successful sibling runs as comparisons. Each accepted edit targeted a mode spanning 7.4 tasks on average, with a minimum of three. The behavioral gate rejected three of 16 candidates that passed the score gate. On sealed Test tasks, the four targeted failure patterns in Table 9 decreased by 42% to 65%, compared with 5% for untargeted failures. These analyses document component use and associated behavioral changes.

Given the high cost of approximately USD 7,352 and 128 hours of rollout wall-clock time for four GPT-5 component ablation campaigns (USD 1,838 and 32 hours per campaign), we could not complete these additional ablations during the rebuttal period. We will distinguish the scope of our evidence in the revised paper: complete-framework gains across six backbones and frozen transfer to WindowsAgentArena, optimizer comparisons on Qwen3-VL-32B and GPT-5, and component ablations on Qwen3-VL-32B.

Failure patterns and settings with limited improvement. As discussed in Section 4.3 and Table 9, remaining bottlenecks include incorrect targets or evaluator-relevant states that appear visually valid, limited adaptation flexibility in OpenCUA's native action interface, and persistent long-horizon execution failures in Qwen. We will expand this discussion in the revision.


Thank you again for your constructive feedback. We hope these clarifications and additional results address your concerns and support a more favorable assessment of our paper.

AuthorsOpenReview ↗

Gentle Reminder: Our Rebuttal and Revised Manuscript

Dear Reviewer NjzW,

Thank you again for your detailed feedback. We would appreciate your consideration of Parts 1/3, 2/3, and 3/3 of our response, which address your main concerns as follows:

  • Distinctions from related methods: We compare GUI-HARVEST with HarnessLens and HarnessEvolve, highlighting joint visual diagnosis of repeated runs of the same task and verification of behavioral predictions recorded before evaluation, with task scores hidden from the validator.
  • Repeatability and cumulative improvement: An independent Qwen3-VL-32B restart reaches 52.62% Test versus 53.18% originally. Evaluated after optimization, all seven accepted updates improve held-out Test performance. Patch-removal and standalone-patch analyses further examine contributions and interactions.
  • Validation reliability: We report coverage across 770 judgments, 88.3% human agreement on 111 audited judgments, and cross-model replays that preserve all 106 recorded final decisions.
  • Comparison fairness: With matched candidate counts, rollout budgets, and Search-only score feedback on Qwen3-VL-32B, GUI-HARVEST retains a 2.76-point Test advantage over Meta-Harness, with a 95% CI of [0.9, 5.3].
  • Stronger-backbone evidence: On GPT-5, GUI-HARVEST exceeds Meta-Harness by 3.76 Test points at equal candidate counts, retaining each method’s native selection policy. We also report component-use analyses and reductions in targeted failure patterns.

These additions are included in the revised manuscript. We would welcome any remaining concerns or questions that we could clarify before the discussion closes. Thank you for your time and consideration.

Reviewer_oKVCOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed the GUI-HARVEST, an optimizer that improves the executable harness of GUI agent. The harness includes prompts, action interfaces, control flow, verification, termination). The system consists of four roles 1) Evidence Analyst 2) Cross-task Clusterer, 3) Harness Engineer 4) Validator and they operate in a closed loop with a test set.

Their experiments covers six frozen backbones ((Qwen3-VL-8B/32B, OpenCUA-32B/72B, Gemini 3.1 Pro, GPT-5), all six backbones improve on OSWorld-Verified.

Critical strengths

Principled, GUI-specific diagnosis loop that generalizes: Aligning intent with visual state changes, using repeated executions as a joint evidence unit produces harness changes that transfer to held-out tasks. Registering predictions upfront and validating them separately from score is a good guardrail.

Meaningful evidence of generalization: The paper shows gain on Test for every backbone and outperforms Self-Harness, Meta-Harness etc.

Critical weaknesses

Cost of optimizer: Paper reported the frozen-harness inference costs and often improved, but it gives almost no quantitative information about the total API / compute cost of the optimization process itself.

Beyond OSWorld: Almost all the tasks and edits are tied to OSWorld, no other environment or benchmark frameworks were measured.

Questions for authors
  1. Can you add total API / compute cost of the optimization process itself.
  2. Can you share the details on other benchmark and add more backbones like GLM or Kimi
2 comments
AuthorsOpenReview ↗

Response to Reviewer oKVC (Part 1/2)

Thank you for your positive assessment and constructive suggestions. We appreciate your recognition of our GUI-specific diagnosis and the value of recording behavioral predictions before evaluation.


Response to W1 and Q1

Q1. Can you add total API / compute cost of the optimization process itself.

Optimization costs. Following your suggestion, we compiled the costs of the original optimization campaigns from execution and usage logs. Each evaluated harness uses 80 Search tasks, 80 Validation tasks, three executions per task, and a 15-step limit. Rollout totals include the initial harness and all evaluated candidates.

Target backboneEvaluated candidatesTotal rolloutsTarget-model tokens (input / output)Target-model API (USD)Optimizer tokens (input / output)Optimizer API (USD)Cumulative VM hoursVM cost (USD)Rollout wall-clock time (h)Total cost (USD)
Gemini 3.1 Pro189,1201.11B / 53M2,67251.7M / 3.3M711,29565222,808
GPT-5168,1601.26B / 25M1,67346.6M / 3.4M701,91095321,838
Qwen3-VL-32B2512,4804.94B / 73MSelf-hosted77.9M / 6.1M1263,73018662312 + GPU
Qwen3-VL-8B168,1602.86B / 27MSelf-hosted50.8M / 4.7M921,3586823160 + GPU
OpenCUA-72B178,6400.75B / 10MSelf-hosted48.7M / 3.9M791,0735418133 + GPU
OpenCUA-32B147,2000.67B / 34MSelf-hosted40.3M / 3.4M691,0855418123 + GPU

B and M denote billion and million tokens. Optimizer charges include all four Claude Sonnet 5 roles and prompt caching. VM hours are cumulative, while wall-clock time covers rollouts only. For GPT-5 and Gemini 3.1 Pro, optimizer API charges account for only 3% to 4% of the total cost of an optimization run.

Harness reuse. Optimization is a one-time expense, after which the frozen harness can be reused in all future deployments. Figure 9 shows that GPT-5 and Gemini 3.1 Pro improve accuracy while reducing inference API costs. For Qwen3-VL-32B, full-suite API cost rises from USD 27.3 to USD 34.2, while the score increases from 38.6% to 50.9%. At the 15-step operating point, API cost per successful task falls by approximately 19% for GPT-5, 40% for Gemini 3.1 Pro, and 5% for Qwen3-VL-32B, relative to the initial harnesses. These savings can accumulate as the frozen harness is reused, without repeating optimization.

Amortization. At the operating points reported in Section 4.4, our 15-step GPT-5 harness achieves 62.4% Full accuracy, compared with 62.6% for published Agent S3 at 100 steps, at approximately USD 0.20 vs. USD 0.72 in API cost per task. At similar accuracy, the USD 0.52 per-task API saving amortizes the measured USD 1,838 optimization cost after approximately 3,500 task executions. Counting deployment API calls and this one-time cost, savings reach 47% after 10,000 tasks and 67% after 50,000 tasks relative to that Agent S3 operating point, assuming comparable future workloads and the reported prices. In this case, reusing the GPT-5 harness can recover the upfront optimization cost and yield increasing net savings at similar accuracy.

▼▼ Continued in Part 2/2. ▼▼

AuthorsOpenReview ↗

Response to Reviewer oKVC (Part 2/2)

▲▲ Continued from Part 1/2. ▲▲

Response to W2 and Q2: Other benchmarks

Q2. Can you share the details on other benchmark and add more backbones like GLM or Kimi

Transfer beyond OSWorld. We agree that evaluation in other environments is important. The submission included frozen-harness transfer to WindowsAgentArena (WAA), reported in Tables 11 and 12. The harnesses were optimized on OSWorld and then evaluated on WAA at a 50-step limit, with the following same-backbone comparisons.

BackboneAgent S3 (%)Frozen GUI-HARVEST (%)Improvement (pp)
Qwen3-VL-32B38.2144.68+6.47
GPT-550.8864.75+13.87

As detailed in Appendix G.2, transfer adapts Windows observation, action, and logging interfaces, together with platform-specific prompt and command conventions. The optimized harness is otherwise kept fixed, and no WAA trajectory is used for optimization or harness selection. These results support transfer across operating systems, applications, and task distributions. We will make this evidence more prominent in the revision.

Why optimization uses OSWorld. OSWorld is a widely used benchmark for desktop GUI agents, adopted by representative GUI agents such as Agent S3, OS-Symphony, and OpenCUA. OSWorld-Verified addresses more than 300 task and evaluation issues and allows easy parallel cloud execution, supporting reliable and efficient repeated evaluation during optimization. We therefore optimized on OSWorld and used WAA for independent transfer evaluation. We leave direct harness optimization on additional platforms to future work.


Response to Q2: Additional backbones

Additional GLM experiment. Following your suggestion, we completed harness optimization for GLM-5.3-Flash using the same Agent S3 initial harness, 80/80/201 Search/Validation/Test split, 15-step limit, and K = 3 executions per task. The selected harness achieves 71.39% on Test, up from 66.95%, and 72.55% on Full, up from 67.66%.

GLM-5.3-Flash harnessSearch (80)Validation (80)Test (201)Full (361)
Initial harness67.3769.7466.9567.66
GUI-HARVEST71.8276.1871.3972.55

Domain results show gains across all five groups, with the largest improvements in professional applications (+14.3 points), OS tasks (+8.3), and Office (+6.0).

DomainTasksInitial harness (%)GUI-HARVEST (%)Gain (pp)
OS2479.1787.50+8.3
Office11773.4779.45+6.0
Daily7866.2667.69+1.4
Professional4967.3581.63+14.3
Multi-application9358.7259.29+0.6

Failure patterns and learned mechanisms. Like Gemini 3.1 Pro and GPT-5, GLM can declare completion without verifying the saved artifact, relying on a toolbar value or a code-reported edit count. A characteristic failure is substituting an existing object when the requested one is missing. For example, GLM copied a desktop folder in place of the undefined $sourceDir. The optimized harness adds feasibility checks based on task text, prompts to verify the saved file and read back specified objects before completion, and a headless LibreOffice save round-trip for xlsx, docx, and pptx files modified during a code-agent session. Plan-format checks also avoid invoking the code agent.

A full Kimi optimization campaign and held-out evaluation require additional time and compute. We leave this extension to future work.


Thank you again for your constructive feedback. We hope the optimization-cost breakdown, cross-platform transfer, and additional GLM results address your concerns.

Reviewer_KLkjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

GUI-HARVEST automatically improves the executable harness around a frozen GUI model. It compares repeated executions of the same task using aligned actions and before-and-after screenshots, aggregates recurring failures across tasks, and translates them into source-code edits. Each edit records behavioral predictions before evaluation and must pass both score and behavior checks. Experiments on OSWorld-Verified cover six backbones, with additional frozen-harness transfer to WindowsAgentArena.

Critical strengths
  1. The method addresses concrete difficulties in GUI harness adaptation. A model can report completion while the interface remains unresolved, and repeated runs can expose different paths for the same task. Comparing actions and visual transitions across these runs provides a sensible basis for diagnosis. The example in Appendix I connects an uncommitted edit to a reusable termination guard, making the proposed evidence-to-edit process understandable.

  2. The evaluation provides evidence beyond the optimization tasks. All six backbones improve on the sealed OSWorld-Verified Test split; Qwen3-VL-32B improves from 43.02% to 53.18%. The frozen WAA transfer gains of 6.47 and 13.87 percentage points for Qwen3-VL-32B and GPT-5 are also relevant, since reusable harness changes could otherwise specialize to the search environment.

Critical weaknesses
  1. The contribution of GUI-specific diagnosis is not fully isolated.

Appendix G.8 matches the initial harness, optimizer model and round cap, but does not establish matched search resources. GUI-HARVEST evaluates 25 candidates for Qwen3-VL-32B (Appendix D.2), whereas Meta-Harness proposes two per round. Their selection feedback also differs: GUI-HARVEST uses aggregate Search and Validation scores for promotion; Meta-Harness selects by Search score, with Validation withheld. Retaining each method's selection policy is reasonable, but these differences limit attribution of the advantage specifically to GUI-aware diagnosis.

The K=1 ablation in Table 3 also reduces the number of available rollouts while removing within-task trajectory comparisons. All selected harnesses receive the same K=3 final evaluation (Appendix G.4), so the concern is evidence during optimization. This ablation supports the value of repeated evidence as a whole, but does not isolate contrastive diagnosis from additional evidence. The protocol should also clarify whether K=1 affects only diagnostic evidence or the repeat count used to estimate promotion scores.

  1. Evaluation uncertainty and optimization uncertainty remain insufficiently characterized.

For fixed-harness evaluation, Appendix C analyzes repeat counts and standard errors, but the main differences lack paired uncertainty estimates. These would be useful for the 3.82-point Test advantage over hard-gate-only, the 5.78-point advantage over Meta-Harness, and smaller gains such as OpenCUA-32B's 1.49 points on 201 Test tasks. Existing task-level outcomes could support paired intervals, with repeated runs grouped by task.

A separate question is whether independently restarting the optimization discovers comparably effective harnesses. Three executions per task under a fixed harness do not answer this. Independent optimization repetitions are not reported. Given their potential cost, I would prioritize uncertainty analysis using existing outcomes and clarification of any repetitions already performed; if none were performed, the claims about optimization stability should be qualified.

  1. The aggregate reliability of behavioral validation is not characterized.

The deterministic checks establish evidence consistency, while semantic judgments remain model based. Under the aggregation rule, one supported task with all other tasks inconclusive and none contradicted can pass a prediction. This need not be inappropriate when triggers are absent, but actual coverage matters. Inconclusive cases should distinguish absent triggers, missing evidence and ambiguous behavior. All four roles use Claude Sonnet 5, so errors in proposing and judging an edit could be correlated. A small independent human audit of supported and contradicted verdicts would help assess judgment quality.

Questions for authors
  1. Please report actual candidate, rollout and optimizer-token budgets and clarify the difference in validation feedback during selection. Could existing records or a targeted control help separate diagnosis from selection effects? For repeated evidence, would a K=3 control with independent per-run diagnosis followed by pooled findings, without later joint access to the raw trajectories, help isolate contrastive diagnosis? Candidate-evaluation budgets, promotion rules and analysis budgets should be controlled.

  2. Could the authors provide task-level paired uncertainty estimates from existing outcomes and clarify whether independent optimization repetitions were performed? If not, please state the resulting limitation on claims about optimization stability.

  3. Please report supported, contradicted and inconclusive rates, separating absent triggers from missing or ambiguous evidence. On a small sample independently assessed by human reviewers, how well do supported and contradicted verdicts agree with the recorded behavior?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KLkj (Part 1/3)

Thank you for your careful review and concrete suggestions. We appreciate your recognition of our diagnostic approach and the evidence from held-out evaluation and transfer to WindowsAgentArena.


Response to W1 and Q1: Optimization budgets and selection feedback

Q1. Please report actual candidate, rollout and optimizer-token budgets and clarify the difference in validation feedback during selection. Could existing records or a targeted control help separate diagnosis from selection effects?

Original comparison. As described in Appendix G.8, the methods shared the initial harness, optimizer model (Claude Sonnet 5), and ten-round cap while retaining their native policies. Meta-Harness proposes two candidates per round and selects by Search score. GUI-HARVEST evaluates up to five candidates per round and uses aggregate Search/Validation scores plus behavioral validation. Validation trajectories are hidden from the Evidence Analyst and Harness Engineer. We agree that the shared round cap does not imply equal evaluation resources.

Additional controls and actual budgets. Following your suggestion, we extended Meta-Harness under its native policy to 25 evaluated candidates and evaluated GUI-HARVEST without Validation score feedback. All configurations below use Qwen3-VL-32B, three executions per task, and a 15-step limit. Search-only evaluations use the same 80 Search tasks per harness, while Full GUI-HARVEST also evaluates 80 Validation tasks. Rollout totals include the initial harness and all evaluated candidates.

ConfigurationEvaluated candidatesTotal rolloutsOptimizer tokens (input / output)Optimizer API (USD)Score feedbackTest (%)
Meta-Harness, original205,040140.7M / 0.55M48Search only47.40
Meta-Harness, extended256,240219M / 0.71M73Search only47.92
GUI-HARVEST without Validation256,24079.6M / 5.9M123.3Search only (behavioral gate retained)50.68
Full GUI-HARVEST2512,48077.9M / 6.1M126Search + Validation53.18

Optimizer API charges account for prompt caching.

Results. Extending Meta-Harness from 20 to 25 candidates raises its Test score from 47.40% to 47.92%, a 0.52-point gain. At 25 candidates, Full GUI-HARVEST achieves 53.18%, exceeding extended Meta-Harness by 5.26 points. When both methods use 25 candidates, 6,240 rollouts, and Search-only score feedback, GUI-HARVEST without Validation achieves 50.68%, compared with 47.92% for extended Meta-Harness. This gives GUI-HARVEST a 2.76-point advantage, with a paired 95% confidence interval of 0.9 to 5.3 points. Validation feedback adds a further 2.50 percentage points. Although GUI-HARVEST incurs higher optimizer costs (USD 123.3 vs. USD 73), we consider the additional expense justified by its stronger Test performance. Moreover, for GPT-5 and Gemini 3.1 Pro, optimizer costs account for only approximately 3% to 4% of total campaign costs (see our response to Reviewer oKVC on W1 and Q1 for details), and reusing the frozen harness amortizes the one-time optimization cost across subsequent tasks.

Evidence from existing records. GUI-HARVEST selected its final harness at candidate 20, and the remaining five candidates were rejected. The final harness therefore did not depend on these five extra candidate evaluations. Replaying the optimization record also shows that Validation rejected only two of nine candidates that improved Search scores and passed the behavioral gate. Thus, Validation changed only a small number of otherwise eligible promotion decisions.

These results show that additional candidates and Validation score feedback do not fully explain the advantage of GUI-HARVEST. We will clarify the budgets and selection protocol, and add these new results after revision.

▼▼ Continued in Part 2/3. ▼▼

AuthorsOpenReview ↗

Response to Reviewer KLkj (Part 2/3)

▲▲ Continued from Part 1/3. ▲▲

Response to W1 and Q1: Joint diagnosis versus additional evidence

Q1. For repeated evidence, would a K=3 control with independent per-run diagnosis followed by pooled findings, without later joint access to the raw trajectories, help isolate contrastive diagnosis? Candidate-evaluation budgets, promotion rules and analysis budgets should be controlled.

Clarifying K = 1. In Table 3 of the paper, K = 1 applies to both diagnostic evidence and the executions used to estimate promotion scores. All selected harnesses subsequently receive the same K = 3 final evaluation. We agree that this ablation changes both evidence quantity and scoring precision, so it does not isolate joint diagnosis. We will clarify this protocol in the revision.

Independent diagnosis control. Following your suggestion, we evaluated a control that analyzes each run separately before pooling the findings. It uses three executions per task, K = 3 promotion scoring, a 25-candidate cap, and the same promotion rule as the full method. Each analyst sees only one run. Failed runs produce findings, while successful runs produce summaries of their successful paths, including step references and observed screen effects. These records undergo deterministic verification before being pooled for the Clusterer, preserving information from successful runs.

Qwen3-VL-32B configurationDiagnostic evidenceK for promotion scoringTest (%)Full (%)
K = 1 ablation, original Table 3One run per task147.4443.12
Independent per-run diagnosis, then poolingThree runs analyzed separately350.6647.52
GUI-HARVEST with joint diagnosisThree runs compared jointly353.1850.94

Joint diagnosis improves Test by 2.52 points and Full by 3.42 points over independent diagnosis. Both variants use the same candidate-evaluation budget and promotion rules, with comparable campaign optimizer costs (USD 126 for joint diagnosis vs. USD 133 for independent diagnosis). Following your suggestion, downstream stages of the independent variant use pooled findings without joint access to the original raw trajectories.

Diagnostic quality. Some findings incorrectly attributed failure to an action that also occurred in a successful run of the same task under the same screen state. Such findings accounted for 31% with independent per-run diagnosis, compared with 8% with joint diagnosis. These results support the value of comparing successful and failed executions to identify differences relevant to the outcome.


Response to W2 and Q2

Q2. Could the authors provide task-level paired uncertainty estimates from existing outcomes and clarify whether independent optimization repetitions were performed?

Paired uncertainty estimates. Following your suggestion, we used the existing outcomes on the 201 Test tasks, resampling tasks jointly across methods and keeping each task's repeated executions together. Differences and intervals below are in percentage points.

ComparisonMean difference95% confidence interval
GUI-HARVEST vs. Meta-Harness, original comparison+5.781.9 to 9.6
GUI-HARVEST vs. hard-gate-only, original comparison+3.821.1 to 6.6
OpenCUA-32B, optimized vs. initial harness+1.49-0.5 to 3.5

These intervals quantify evaluation uncertainty for the selected harnesses. The first two intervals lie entirely above zero, providing statistical support for GUI-HARVEST's gains over Meta-Harness and hard-gate-only. OpenCUA-32B shows a small observed improvement, although its 95% confidence interval includes zero. This modest gain is consistent with the limited adaptation flexibility of its native action interface discussed in Section 4.3.

Independent optimization restart. Following your suggestion, we added an independent Qwen3-VL-32B optimization with the same data split, initial harness, and budget, only changing the optimizer roles' sampling seeds.

Optimization runTest (%)Gain over initial harness (pp)Gain over Meta-Harness with 25 candidates (pp)
Original53.18+10.16+5.26
Additional restart52.62+9.60+4.70

Both runs independently discovered the same four types of harness edits: commit checks, routing between GUI and code execution, retirement of ineffective routes, and screen-difference evidence. The similar scores and recurring mechanisms support repeatability in this setting.

▼▼ Continued in Part 3/3. ▼▼

AuthorsOpenReview ↗

Response to Reviewer KLkj (Part 3/3)

▲▲ Continued from Part 2/3. ▲▲

Response to W3 and Q3

Q3. Please report supported, contradicted and inconclusive rates, separating absent triggers from missing or ambiguous evidence. On a small sample independently assessed by human reviewers, how well do supported and contradicted verdicts agree with the recorded behavior?

Behavioral-validation coverage. Each task-level validation judgment assesses one behavioral prediction on one target task. We analyzed 770 judgments across all six backbones.

BackboneJudgmentsSupportedContradictedInconclusiveCandidates rejected only by the behavioral gate
Qwen3-VL-32B18763%21%16%5
Qwen3-VL-8B11858%24%19%3
Gemini 3.1 Pro13966%19%15%4
GPT-512664%20%16%3
OpenCUA-72B11255%22%22%2
OpenCUA-32B8852%23%25%2
Total77061%21%18%19

Percentages are rounded. Overall, 82% of judgments are conclusive, and the behavioral gate rejects 19 candidates that the score gate would accept. As requested, we separate inconclusive cases into absent triggers, missing evidence, and ambiguous behavior. The results are below.

BackboneRelevant trigger absentMissing evidenceAmbiguous behavior
Qwen3-VL-32B53%17%30%
Qwen3-VL-8B59%18%23%
Gemini 3.1 Pro48%14%38%
GPT-550%15%35%
OpenCUA-72B72%12%16%
OpenCUA-32B82%9%9%
Total61%14%25%

Predictions with one supported task and all others inconclusive. Among 66 predictions associated with accepted harness edits, only 5 had one supporting task judgment and all others inconclusive because triggers were absent. 3 came from OpenCUA, whose action-normalization edits trigger on relatively few tasks. Removing the 2 corresponding Qwen3-VL-32B edits individually from the final harness reduced Test scores by 1.1 and 0.4 points. Thus, this acceptance pattern is uncommon, and the corresponding examined edits still contribute to held-out performance.

Human verification. We sampled 18 complete candidate patches from all 6 original backbones, covering promoted and rejected candidates. The sample includes all 35 required predictions and 111 prediction–task judgments for those patches. An annotator reviewed the frozen predictions, pre-edit findings, and post-edit executions, with model verdicts, patches, and task scores withheld.

Sonnet 5 judgmentReviewed judgmentsConfirmed by human assessmentAgreement
Supported706491.4%
Contradicted232087.0%
Inconclusive181477.8%
Total1119888.3%

For the 18 audited harness edits, applying the original aggregation rule to the human task-level verdicts reproduced the behavioral gate’s pass/reject decisions for 17/18 edits (94.4%). The disagreement occurred in a candidate already rejected by the score gate. Substituting human labels preserved the final promote/rollback decisions for all 18 sampled candidates.

Full-campaign cross-model verification. To address the concern about correlated errors from using Claude Sonnet 5 for both proposing and judging harness edits, we used GPT 6.1 Sol and Kimi K3 to separately reassess all 770 prediction–task judgments, covering 231 predictions in 106 candidate patches. They received the same task-local evidence as the original Validator, with existing judgments, patches, and scores withheld.

ValidatorAgreement with Sonnet 5Changed promote/rollback decisions
GPT 6.1 Sol711/770 (92.3%)0/106
Kimi K3686/770 (89.1%)0/106

For each model, we replayed the recorded candidates using the original per-prediction aggregation rule and score gate. Both cross-model checks preserved all recorded promote/rollback decisions. We will include these verification results in the revision.


Thank you again for your constructive suggestions. The additional controls, paired uncertainty estimates, optimization restart, and validation audits provide more direct evidence for the mechanisms you identified. We hope these results address your concerns.

AuthorsOpenReview ↗

Gentle Reminder: Our Rebuttal and Revised Manuscript

Dear Reviewer KLkj,

Thank you again for your thoughtful feedback. We would appreciate your consideration of Parts 1/3, 2/3, and 3/3 of our response, which address your concerns through the following clarifications and additional results:

  • Evaluation budgets and selection feedback: On Qwen3-VL-32B, GUI-HARVEST retains a 2.76-point Test advantage over Meta-Harness when both use 25 candidates, 6,240 rollouts, and Search-only score feedback.
  • Joint diagnosis: With three executions per task, the same promotion rule, and matched candidate-evaluation budgets, joint diagnosis outperforms independent per-run diagnosis followed by pooling by 2.52 Test points.
  • Uncertainty and repeatability: We report task-paired confidence intervals and an independent Qwen3-VL-32B restart, which reaches 52.62% Test versus 53.18% originally and discovers similar edit mechanisms.
  • Validation coverage: Across 770 judgments, 82% are conclusive. We distinguish absent triggers, missing evidence, and ambiguous behavior, and examine predictions supported by only one task.
  • Independent audits: Human agreement is 88.3% across 111 task-level judgments. Aggregating human verdicts reproduces 17/18 behavioral-gate decisions, while alternative-validator replays preserve all 106 recorded final decisions.

These additions are included in the revised manuscript. We welcome any remaining questions and would be happy to clarify them before the discussion closes. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Paper Following Reviewer Feedback

Dear Reviewers,

Thank you again for your constructive feedback. We have uploaded a revised manuscript incorporating the additional experiments, analyses, and clarifications presented in our rebuttal. Revised material is highlighted in blue.

We would appreciate your consideration of our rebuttal responses alongside the updated manuscript, and we welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.