- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes GUI-HARVEST, a method for improving GUI agents by editing their execution harness while keeping the backbone model fixed. It analyzes multiple multimodal execution traces, clusters failure patterns across tasks, proposes targeted harness modifications, and validates candidate modifications using both measured utility and a soft check of whether predicted behavioral changes actually occurred. The experiments use an 80/80/201 Search/Validation/Test split of OSWorld-Verified, six backbone models, comparisons with other harness optimization methods, ablations, and transfer tests on WindowsAgentArena. On the held-out 201-task OSWorld-Verified Test split, Qwen3-VL-32B improves from 43.02% to 53.18% (+10.16 percentage points), with improvements also reported for the other evaluated backbones.
Critical strengths
-
Meaningful evaluation beyond the harness-development split. The method is tested on a separate 201-task set with six different backbones, which helps distinguish general improvements from merely memorizing the tasks used to edit the harness. The reported +10.16 percentage-point result for Qwen3-VL-32B is a substantial effect on this held-out set.
-
A concrete, analyzable optimization mechanism. Instead of treating failed GUI executions as undifferentiated feedback, the method combines visual/action traces, repeated trials, shared failure patterns, and targeted harness patches. Ablations and WindowsAgentArena transfer results provide useful, although not definitive, support for this design.
Critical weaknesses
-
The cost evidence does not establish end-to-end cost efficiency. The reported inference/API cost savings concern evaluation with a frozen, optimized harness. Appendix G.10 explicitly excludes optimization-process costs. The search and validation loops, repeated environment executions, and model-based analysis may be expensive enough to change the practical cost comparison substantially. This matters because lower operating cost is an important part of the paper's appeal.
-
The behavioral-gating mechanism and optimization stability are not fully isolated. The soft gate relies on a model's judgment about predicted behavioral changes, yet its semantic accuracy is not independently established. The ablations do not fully equalize optimization trajectories and budgets, and a single search/validation/test split does not characterize variability across independent optimization runs. Consequently, the data support the overall pipeline more strongly than the precise causal contribution of the gate.
Questions for authors
-
What is the complete cost of one optimization run, including model tokens/API charges, environment or VM hours, and repeated rollouts? How many subsequent evaluation or deployment tasks are required to amortize this cost?
-
On a human-audited sample of proposed harness patches, how often does the soft gate correctly accept or reject the predicted behavioral change? Does it outperform a score-only gate at a matched proposal and execution budget?
-
Can the authors report a repeated optimization run or a different split for at least one backbone, together with task-paired uncertainty on the final held-out Test improvement?
3 comments
Response to Reviewer C5L7 (Part 2/2)
▲▲ Continued from Part 1/2. ▲▲
Response to W2 and Q2
Q2. On a human-audited sample of proposed harness patches, how often does the soft gate correctly accept or reject the predicted behavioral change? Does it outperform a score-only gate at a matched proposal and execution budget?
Human audit of the soft gate’s accept/reject decisions. We sampled 18 candidate patches across all 6 original backbones, including all 35 required predictions and 111 prediction–task judgments for those patches. An annotator reviewed the frozen predictions, pre-edit findings, and post-edit executions, with model verdicts, patches, and scores withheld. Task-level agreement with Sonnet 5 was 98/111 (88.3%). For the 18 audited harness edits, applying the original aggregation rule to the human task-level verdicts reproduced the behavioral gate’s pass/reject decisions for 17/18 edits (94.4%). The breakdown is shown below.
| Original behavioral-gate decision | Reviewed candidates | Confirmed by human-label aggregation |
|---|---|---|
| Pass | 11 | 10 |
| Reject | 7 | 7 |
| Total | 18 | 17 (94.4%) |
The behavioral-gate disagreement was in a candidate already rejected by the score gate. Therefore, replacing the model verdicts with human verdicts left the final accept/reject decisions unchanged for all 18 audited harness patches.
Full-campaign cross-model verification. To check whether changing the validator model would alter the recorded promote/rollback decisions, we used GPT 6.1 Sol and Kimi K3 to separately reassess all 770 judgments from 231 predictions across 106 candidates. Their agreement with Sonnet 5 was 92.3% and 89.1%, respectively. Replaying each model's labels under the original aggregation and score gates preserved all recorded promote/rollback decisions. Task-level verification tables appear in our response to Reviewer KLkj (Part 3/3, W3 and Q3).
Comparison at matched budgets. We extended the score-only run from 21 to 25 evaluated candidates. All four additional candidates were rejected, leaving its selected harness unchanged. Both variants use the same Search/Validation score gate and 12,480 rollouts, including the initial harness.
| Qwen3-VL-32B result | Score-only gate | Score + behavioral gates |
|---|---|---|
| Evaluated candidates | 25 | 25 |
| Test (%) | 49.36 | 53.18 / 52.62 (two runs) |
Both full-method runs outperform score-only. Their mean advantage over the score-only run is 3.54 points, with a task-paired 95% confidence interval of 1.3 to 5.8 points. These results show that, for Qwen3-VL-32B, adding the behavioral soft gate improves held-out Test performance over score-only gating at matched proposal and execution budgets.
Response to W2 and Q3
Q3. Can the authors report a repeated optimization run or a different split for at least one backbone, together with task-paired uncertainty on the final held-out Test improvement?
Independent repeated optimization run. Following your suggestion, we added an independent repeated optimization run using Qwen3-VL-32B, with the same data split, initial harness, and budget, changing only the optimizer roles' sampling seeds.
| Optimization run | Test (%) | Full (%) | Test gain over initial harness (pp) |
|---|---|---|---|
| Original | 53.18 | 50.94 | +10.16 |
| Additional repeated run | 52.62 | 50.47 | +9.60 |
The repeated run's Test score is only 0.56 points below the original and remains above all reported baselines. Both runs independently discovered the same four types of edits: commit checks, routing between GUI and code execution, retirement of ineffective routes, and screen-difference evidence. The similar held-out gains and recurring mechanisms support repeatability.
Task-paired uncertainty. We jointly resampled the 201 Test tasks across the initial and optimized harnesses, keeping each task's three executions together. Differences and 95% confidence intervals below are in percentage points.
| Backbone | Test improvement | 95% confidence interval |
|---|---|---|
| Qwen3-VL-32B | +10.16 | 5.8 to 14.5 |
| Qwen3-VL-8B | +6.05 | 2.2 to 9.9 |
| Gemini 3.1 Pro | +6.20 | 2.6 to 9.8 |
| GPT-5 | +5.51 | 1.9 to 9.1 |
| OpenCUA-72B | +2.93 | 0.8 to 5.1 |
| OpenCUA-32B | +1.49 | -0.5 to 3.5 |
All six backbones show positive observed improvements. The intervals quantify evaluation uncertainty for the selected harnesses, with five lying entirely above zero. OpenCUA-32B's modest gain is consistent with the limited adaptation flexibility of its native action interface discussed in Section 4.3. We will add the paired estimates and independent restart results to the revision.
Thank you again for your constructive feedback. We hope the cost accounting, gate audits, matched-budget comparison, and independent restart address your concerns.
Gentle Reminder: Our Rebuttal and Revised Manuscript
Dear Reviewer C5L7,
Thank you again for your careful review. We would appreciate your consideration of Parts 1/2 and 2/2 of our response, which provide the following additional evidence:
- Optimization costs and amortization: We report token/API usage, rollout counts, and VM resources. The GPT-5 campaign costs USD 1,838. At the reported operating points, API savings over Agent S3 recover this cost after approximately 3,500 tasks, assuming comparable workloads and prices.
- Human and cross-model audits: Human verdicts reproduce 17/18 behavioral-gate decisions (94.4%), while all 18 final promote/rollback decisions remain unchanged. Alternative-validator replays also preserve all 106 recorded final decisions.
- Matched-budget gate comparison: With 25 candidates and 12,480 rollouts per campaign, the two full-method runs average 3.54 Test points above score-only gating on Qwen3-VL-32B, with a task-paired 95% CI of [1.3, 5.8].
- Optimization repeatability and uncertainty: An independent Qwen3-VL-32B restart reaches 52.62% Test versus 53.18% originally and recovers similar edit mechanisms. We also provide task-paired confidence intervals for all six original backbone gains, with five lying entirely above zero.
These additions are included in the revised manuscript. We welcome any remaining questions and would be happy to clarify them before the discussion closes. Thank you for your time and consideration.
Response to Reviewer C5L7 (Part 1/2)
Thank you for your careful review and concrete suggestions. We appreciate your recognition of our held-out evaluation and the inspectable optimization mechanism.
Response to W1 and Q1
Optimization costs. We agree that the frozen-harness inference costs in Section 4.4 and Appendix G.10 do not include optimization costs. Following your suggestion, we compiled the costs of the original campaigns from execution and usage logs. Each evaluated harness uses 80 Search tasks, 80 Validation tasks, three executions per task, and a 15-step limit. Rollout totals include the initial harness and all evaluated candidates.
B and M denote billion and million tokens. Optimizer charges include all four Claude Sonnet 5 roles and prompt caching. VM hours are cumulative, while wall-clock time covers rollouts only. For GPT-5 and Gemini 3.1 Pro, optimizer API charges account for only 3% to 4% of the total cost of an optimization run.
Amortization. At the operating points reported in Section 4.4, our 15-step GPT-5 harness achieves 62.4% Full accuracy, compared with 62.6% for published Agent S3 at 100 steps, at approximately USD 0.20 vs. USD 0.72 in API cost per task. At similar accuracy, the USD 0.52 per-task API saving amortizes the measured USD 1,838 optimization cost after approximately 3,500 task executions. Counting deployment API calls and this one-time cost, savings reach 47% after 10,000 tasks and 67% after 50,000 tasks relative to that Agent S3 operating point, assuming comparable future workloads and the reported prices. In this case, reusing the GPT-5 harness can recover the upfront optimization cost and yield increasing net savings at similar accuracy.
Cost per successful task. At the 15-step operating point in Figure 9, the frozen harness optimized by GUI-HARVEST reduces API cost per successful task by approximately 19% for GPT-5, 40% for Gemini 3.1 Pro, and 5% for Qwen3-VL-32B, relative to the initial harness. For Qwen3-VL-32B, the full-suite cost rises from USD 27.3 to USD 34.2, but accuracy increases from 38.6% to 50.9%, explaining the lower cost per success. These findings show that the optimized harnesses reduce the API cost per successful task during deployment.
▼▼ Continued in Part 2/2. ▼▼