- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines program-level evolution for single-step GUI grounding. It introduces EvoHarness to search over inference programs on synthetic GUI data and evaluates a selected program on ScreenSpot. Comparisons with direct-inference controls show that substantial gains over the seed do not consistently translate into better performance after transfer to matched backends. Repeated executions of unchanged programs also reveal apparent improvements caused by execution variability. The paper proposes a paired acceptance rule and evaluation practices to address these concerns.
Critical strengths
- The selected program shows some transfer beyond the synthetic selection set. Search uses only 40 synthetic GUI examples, yet the selected program achieves 89.15% accuracy on all 1,272 ScreenSpot examples, none of which participated in search. This provides evidence that the program can handle real interfaces under its original execution configuration, extending its demonstrated effectiveness beyond the synthetic data.
- The experiments quantify how execution variability affects candidate selection. Across 20 pairs of repeated executions of identical code, the gain-only rule accepts five apparent improvements, whereas the paired gate accepts none. This directly connects measurement instability to selection decisions and provides useful evidence about how apparent gains enter the search process and how sensitive different acceptance rules are to them.
Critical weaknesses
- The significance of the research question is unclear, and the analysis is limited to a single task with limited representativeness. The paper studies and critically examines program-level evolution for GUI grounding, but does not clearly explain why this particular task is chosen among the many agent tasks available. The stated motivations, such as the influence of surrounding programs on performance and the possibility of automatically optimizing them, apply broadly to agent tasks and do not specifically justify the choice of GUI grounding. The scope of the investigation is also limited. The paper considers static, single-step localization, where execution mainly involves image processing, model calls, and coordinate parsing. This setting does not cover important functions of more complex harnesses, such as planning, state management, tool coordination, and failure recovery. Coding and complex workflow tasks more directly involve these functions and the potential value of evolving them. Observations from this single task, with its relatively simple execution structure, provide a limited basis for broader conclusions about program-level evolution.
- The main conclusions offer limited new insight, leaving the contribution to program-level evolution unclear. The paper’s main conclusions are that outperforming the seed does not imply outperforming a reasonable direct-inference baseline; an increase in score may reflect execution variability rather than program improvement; and a successful evolved artifact alone does not establish the reliability of the search procedure. These conclusions largely follow established evaluation principles and are not particularly surprising. The additional evidence mainly demonstrates these phenomena in the paper’s GUI grounding experiments. The analysis remains confined to this task and the authors’ own evolution pipeline, without showing how the findings inform other program-level evolution methods. The direct baselines, repeated measurements, and paired tests likewise primarily apply existing evaluation methods to this pipeline. Thus, while the paper documents several noteworthy experimental observations, it offers limited new understanding and a limited overall contribution to program-level evolution.
- Some central problems arise primarily from the authors’ own experimental setup rather than common practice in existing agent-evolution research. The paper presents as an important finding that the evolved program outperforms the seed but still falls behind direct inference. However, the seed may contain a mismatch between coordinate outputs and parsing: its parser treats values outside the unit interval as pixel coordinates, whereas four inspected failure responses used a coordinate scale of [0,1000). On the matched Qwen backend, this seed achieves only 20.33% accuracy, compared with 70.67% for Direct point, which also requires just one model call. Starting from such a weak and potentially flawed program, finding that the evolved result still underperforms direct inference primarily points to a problem with the chosen initial program. The paper provides no evidence that existing program-level evolution studies commonly start from programs with such defects. This finding therefore says more about the authors’ own setup than about the research direction as a whole.
- The central comparison mixes evolution performance with cross-backend transfer, making the cause of the evolved program’s disadvantage unclear. The original search uses a two-model execution stack consisting of a stronger model and a cheaper model. One of the champion’s important strategies is to call the cheaper model first and route subsequent calls according to confidence. However, the experiments supporting the finding that the evolved program underperforms direct inference transfer the frozen program to subsequent Qwen and Kimi backends, mapping both original model aliases to the same model. This changes the execution conditions under which the program was searched and selected, as well as the practical role of its routing strategy. The observed disadvantage could therefore arise from the evolved strategy itself or from a mismatch between the program and the new model or interface configuration. The paper reports neither direct-inference controls on the original two-model stack nor evolution rerun under the subsequent configurations. The results establish that this frozen program does not outperform direct-point prediction after transfer; they do not clearly establish whether program-level evolution under the same execution configuration can yield gains over direct inference.
Questions for authors
See Weaknesses.