acceptodds
Withdrawn from ICLR 2027

Rethinking Agent Evolution: Do Evolved GUI Harnesses Actually Outperform Direct Inference?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Program evolution is increasingly used to improve language-model agents, with progress typically measured against the initial program. We study whether seed-relative gains translate into an advantage over well-specified direct interfaces to the same frozen model. We introduce EvoHarness, a framework for evolving GUI grounding harnesses and auditing the resulting gains with frozen interface controls, matched execution backends, and repeated measurement. EvoHarness searches over executable preprocessing, prompting, parsing, and routing code, retaining execution traces for post hoc analysis. In a pilot search, evolution finds a harness that reaches 89.15% accuracy on all 1,272 ScreenSpot examples. Under matched backends, however, the evolved champion does not consistently outperform direct inference controls on the same frozen model: on Qwen, it improves from 20.33% to 54.33% over the seed but remains below a single-call direct-point baseline at 70.67% (paired Holm p=1.26×10^-5); on Kimi, it is not significantly better than either direct control. Repeated evaluation of identical code also yields apparent gains: a gain-only rule accepts 5/20 disjoint same-code pairs, while a paired gate accepts 0/20. These results suggest that seed-relative gains alone are insufficient evidence for agent improvement. We recommend frozen direct-interface controls, matched execution resources, and uncertainty reported at both the program and search levels.

Primary area:
foundation or frontier models, including LLMs
Keywords:
GUI Agents, Program Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iNQVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines program-level evolution for single-step GUI grounding. It introduces EvoHarness to search over inference programs on synthetic GUI data and evaluates a selected program on ScreenSpot. Comparisons with direct-inference controls show that substantial gains over the seed do not consistently translate into better performance after transfer to matched backends. Repeated executions of unchanged programs also reveal apparent improvements caused by execution variability. The paper proposes a paired acceptance rule and evaluation practices to address these concerns.

Critical strengths
  1. The selected program shows some transfer beyond the synthetic selection set. Search uses only 40 synthetic GUI examples, yet the selected program achieves 89.15% accuracy on all 1,272 ScreenSpot examples, none of which participated in search. This provides evidence that the program can handle real interfaces under its original execution configuration, extending its demonstrated effectiveness beyond the synthetic data.
  2. The experiments quantify how execution variability affects candidate selection. Across 20 pairs of repeated executions of identical code, the gain-only rule accepts five apparent improvements, whereas the paired gate accepts none. This directly connects measurement instability to selection decisions and provides useful evidence about how apparent gains enter the search process and how sensitive different acceptance rules are to them.
Critical weaknesses
  1. The significance of the research question is unclear, and the analysis is limited to a single task with limited representativeness. The paper studies and critically examines program-level evolution for GUI grounding, but does not clearly explain why this particular task is chosen among the many agent tasks available. The stated motivations, such as the influence of surrounding programs on performance and the possibility of automatically optimizing them, apply broadly to agent tasks and do not specifically justify the choice of GUI grounding. The scope of the investigation is also limited. The paper considers static, single-step localization, where execution mainly involves image processing, model calls, and coordinate parsing. This setting does not cover important functions of more complex harnesses, such as planning, state management, tool coordination, and failure recovery. Coding and complex workflow tasks more directly involve these functions and the potential value of evolving them. Observations from this single task, with its relatively simple execution structure, provide a limited basis for broader conclusions about program-level evolution.
  2. The main conclusions offer limited new insight, leaving the contribution to program-level evolution unclear. The paper’s main conclusions are that outperforming the seed does not imply outperforming a reasonable direct-inference baseline; an increase in score may reflect execution variability rather than program improvement; and a successful evolved artifact alone does not establish the reliability of the search procedure. These conclusions largely follow established evaluation principles and are not particularly surprising. The additional evidence mainly demonstrates these phenomena in the paper’s GUI grounding experiments. The analysis remains confined to this task and the authors’ own evolution pipeline, without showing how the findings inform other program-level evolution methods. The direct baselines, repeated measurements, and paired tests likewise primarily apply existing evaluation methods to this pipeline. Thus, while the paper documents several noteworthy experimental observations, it offers limited new understanding and a limited overall contribution to program-level evolution.
  3. Some central problems arise primarily from the authors’ own experimental setup rather than common practice in existing agent-evolution research. The paper presents as an important finding that the evolved program outperforms the seed but still falls behind direct inference. However, the seed may contain a mismatch between coordinate outputs and parsing: its parser treats values outside the unit interval as pixel coordinates, whereas four inspected failure responses used a coordinate scale of [0,1000). On the matched Qwen backend, this seed achieves only 20.33% accuracy, compared with 70.67% for Direct point, which also requires just one model call. Starting from such a weak and potentially flawed program, finding that the evolved result still underperforms direct inference primarily points to a problem with the chosen initial program. The paper provides no evidence that existing program-level evolution studies commonly start from programs with such defects. This finding therefore says more about the authors’ own setup than about the research direction as a whole.
  4. The central comparison mixes evolution performance with cross-backend transfer, making the cause of the evolved program’s disadvantage unclear. The original search uses a two-model execution stack consisting of a stronger model and a cheaper model. One of the champion’s important strategies is to call the cheaper model first and route subsequent calls according to confidence. However, the experiments supporting the finding that the evolved program underperforms direct inference transfer the frozen program to subsequent Qwen and Kimi backends, mapping both original model aliases to the same model. This changes the execution conditions under which the program was searched and selected, as well as the practical role of its routing strategy. The observed disadvantage could therefore arise from the evolved strategy itself or from a mismatch between the program and the new model or interface configuration. The paper reports neither direct-inference controls on the original two-model stack nor evolution rerun under the subsequent configurations. The results establish that this frozen program does not outperform direct-point prediction after transfer; they do not clearly establish whether program-level evolution under the same execution configuration can yield gains over direct inference.
Questions for authors

See Weaknesses.

Reviewer_osrwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits GUI harness evolution using direct-inference controls and repeated executions. It finds that substantial improvements over the initial harness do not establish an advantage over strong direct inference.

Critical strengths

The matched-backend comparison provides a useful demonstration of why evaluating against the initial implementation alone can be misleading.

The paper carefully distinguishes operational errors, execution variability, and incomplete search replication. The reported paired statistics are internally consistent.

Critical weaknesses

Direct controls are missing on the original two-model execution stack. Moreover, the matched-backend comparison collapses the champion's two model aliases into one, altering its routing behavior. A coordinate-corrected seed is also needed to separate interface repair from other evolved improvements.

The paired gate suppresses spurious same-code improvements, but its low statistical power on the 40-example selection set may also reject genuine gains. Whether it improves final held-out search performance over gain-only selection remains unestablished.

Questions for authors

How do direct inference and a coordinate-corrected seed compare with the champion on the original execution stack?

Across independent searches, does the paired gate improve final held-out performance over gain-only selection at equal total evaluation cost?

Reviewer_vDmhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper is a "case study" of evolving harnesses for GUI agents where the evolved harness improves on the train set but does not improve on the held out sets. The evolved harness does not outperform baselines of "direct point" and "direct bounding box", and the evolved harness does not even beat the seed harness when evaluated with Kimi K3.

Critical strengths

The paper is technically sophisticated and has a through evaluation. They evaluate with multiple models (Qwen and Kimi), compare to multiple baselines (direct point and direct bbox), and have multiple splits. The paper includes a strategy for rejecting spurious improvements, which they call "ValidGate", which is important to prevent winner's curse when lots of offspring are selected.

Critical weaknesses

First, the writing in this paper is very difficult to understand. Jargon is used without definition, technical details are described haphazardly and with language flourishes that make understanding difficult. In my opinion, pretty much a full rewrite of the paper would be needed to bring the writing to a level of quality appropriate for a published paper. For example, section 4 paragraph 1 describes the splits of data. "SynthGUI procedurally renders seven element classes. The original splits contain 40 examples each: dev and test use a 390×720 light mobile theme, while the dark-theme splits use 720 ×460 web layouts. Median target area is 0.93% of the dev canvas and 0.47% of the held-out dark canvas. Selection uses dev only; test and test dark are held out from mutation and selection. The small synthetic setting exposes program and evaluation behavior rather than representing the full diversity of GUI tasks." This paragraph is very hard to make sense of: how many splits are there total? How many examples are in the dark splits? What are the dark splits used for? The same lack of clarity occurs throughout the paper.

Second, the technical results of the paper are not very valuable. The paper is a "case study" showing that evolution does not beat the baseline due to overfitting on the training set. While there are interesting technical contributions, such as the gating mechanism for rejecting spurious improvements, I am not sure whether the results are substantial enough for a paper.

Questions for authors

Can you explain all of the data splits, training different training runs and model set ups more clearly? I think a full story version of this paper would start with the failed run, show the tricks that improved it, and then finish with results that show positive results (like successful new best harness from evolution). Then, the paper could be presented as a "results from evolving + the tricks needed to make it work".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.