acceptodds
Withdrawn from ICLR 2027

ScreenRSI: Recursive Self-Improvement for GUI Agent Harnesses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern agents combine a large language model (LLM) with a harness that prompts the model and interacts with the environment to complete user tasks. Although this code layer strongly shapes agent behavior, improving it still relies largely on engineers inspecting failures and revising the system by hand. Harness-level recursive self-improvement (HRSI) automates this process by evaluating tasks, analyzing failure modes, and revising the harness over successive rounds while keeping the model and environment fixed. Its central challenge is that sparse, local failure evidence guides changes to shared logic that can affect many other tasks. Successive revisions may therefore accumulate useful behavior or repeatedly break existing capabilities. We study how local repairs can yield broader gains under a limited evaluation budget and whether these gains transfer to unseen tasks. We introduce ScreenRSI, an HRSI framework for GUI agents that uses a semantic task-similarity graph to select failure seeds for coverage and diversity and samples previously successful tasks as preservation tasks. A recurring cycle of task selection, harness repair, and controlled merging turns execution evidence into validated harness revisions. On MobileGym, with Qwen3.8-27B fixed as the acting model and optimization confined to training tasks, the GPT-5.6-Sol optimizer produces a harness with a peak test success rate of 74.77%, improving over the initial harness by 15.71% and reducing false completion by 10.55%. Under the same round budget, graph-guided selection improves test success by 5.73% and reduces false completion by 3.52% relative to random selection. Several key mechanisms for improving the harness emerged during the evolution of ScreenRSI, including state management, stuck detection and recovery, task-specific prompt guidance, and output-format validation. These findings provide insight guidance for designing stronger harnesses.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI Agent, Harness, Recursive Self-Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZAjYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ScreenRSI, a framework for iteratively optimizing GUI agent harnesses while keeping the acting model and environment fixed. It uses a semantic task-similarity graph to select failure tasks for coverage and diversity, together with previously successful tasks for regression protection. An optimizer agent analyzes execution trajectories and modifies the harness, with repair validation and controlled merging producing the next revision. Experiments on MobileGym demonstrate improvements over the initial harness and gains from graph-guided task selection relative to random selection.

Critical strengths
  1. The research question is worthwhile. Treating the harness as an optimization target and exploring how execution feedback can drive iterative revisions to its code and prompts is a promising direction. The work offers a useful perspective on improving GUI agent capabilities with a fixed underlying model.
  2. The graph-guided task selection design is well motivated. Using behavioral similarity to balance failure coverage and diversity, while selecting both nearby and distant successful tasks to check for regressions, addresses the tension between local repair and capability preservation under a limited evaluation budget. The comparison with random selection provides initial empirical support for this design.
Critical weaknesses
  1. The direction of GUI agent harness self-improvement is appealing, but all experiments start from a single custom initial harness. The method is not evaluated on established GUI agent frameworks, such as Agent-S3 [1] or OS-Symphony [2], or on harnesses with different architectures. This limits the broader significance of the results: the gains may depend partly on the particular starting harness, and the evidence does not establish the method’s general applicability to mature GUI agent harnesses.
  2. All experiments are conducted on MobileGym. Given the paper’s broader focus on GUI agent harnesses, evaluation solely in a mobile environment is insufficient to establish applicability across environments. Additional experiments on OSWorld [3] would strengthen the evidence that ScreenRSI generalizes across platforms.
  3. Failure attribution and repair generalization lack empirical analysis. The appendix prompts distinguish failures originating from the harness, model behavior, environment, and evaluator, but the paper does not report their distribution, attribution accuracy, or repair outcomes by source. Since model behavior failures can also trigger harness modifications, the aggregate success-rate gains do not establish whether improvements primarily reflect transferable repairs to harness defects or prompt and control-flow changes that compensate for local model errors.

[1] Gonzalez-Pumariega G, Tu V, Lee C L, et al. The Unreasonable Effectiveness of Scaling Agents for Computer Use. 2025.

[2] Yang B, Jin K, Wu Z, et al. OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026: 22300–22330.

[3] Xie T, Zhang D, Chen J, et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Advances in Neural Information Processing Systems, 2024, 37: 52040–52094.

Questions for authors
  1. What is the provenance of the initial harness, and how does its functionality compare with established GUI agent frameworks? Could the authors apply ScreenRSI to at least one existing framework, or to an enhanced baseline with standard memory, recovery, and validation mechanisms, and report both the additional gains and the customization required for integration?
  2. Can the method transfer to desktop GUI environments such as OSWorld?
  3. In the actual failure analyses, what proportion of failures falls into each source category, and what are the corresponding repair acceptance rates? Were the attributions manually checked? Could the authors report repair success, regressions, and gains on unseen tasks by failure source, and examine whether the final modifications include compensatory rules tailored to particular task patterns?
Reviewer_hNv5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ScreenRSI, a harness-level recursive self-improvement (HRSI) framework for GUI agents that iteratively diagnoses failed trajectories and edits the shared harness code (agent.py, parser.py, prompt.py) while keeping the acting model fixed. To balance local failure repair against global capability regression under a tight evaluation budget, ScreenRSI builds a static semantic task-similarity graph to sample diverse failure seeds alongside nearby and distant successful preservation tasks. On MobileGym with a fixed Qwen3.8-27B acting model, ScreenRSI (using GPT-5.6-Sol as the optimizer) boosts peak test success rate from 59.06% to 74.77% and outperforms random task selection by 5.73%.

Critical strengths
  1. Well-motivated focus on shared harness code evolution: Optimizing the executable harness layer rather than just appending text memories directly addresses real-world agent engineering bottlenecks and surfaces the critical trade-off between local fixes and global regressions.

  2. Principled graph-guided budget allocation: Using a behavior-oriented task-similarity graph to jointly select diverse failure seeds and both semantically close and distant preservation tasks is a clean, lightweight way to guard against regressions without full-suite re-evaluation every round.

  3. Interpretable emergent harness mechanisms: The evolved code organically discovers practical GUI agent design patterns—such as persistent cross-app numeric ledgers, visual stuck/loop detection, and stricter output parsers—which noticeably reduce average execution steps and false completions.

Critical weaknesses
  1. Lack of external baselines and risk of environment-specific overfitting: Despite discussing numerous recent harness-optimization and self-evolving GUI agent methods in Section 2, the paper compares ScreenRSI only against an initial harness and a random-selection ablation. Additionally, Case Study 1 (Figure 6) reveals that the evolved harness hardcodes the benchmark environment's frozen date (Today: Sep. 23, 2026; Last week: Sep. 17–23) into the prompt, contradicting the prompt rules against literal/environment-specific specialization.

  2. Test-set cherry-picking and unstable optimization trajectories: Table 1 and Figure 1 report the peak test-set performance across rounds rather than evaluating a checkpoint selected purely via training/validation signals (e.g., Sol’s reported Train best is Round 6, whereas its reported Test result is cherry-picked from Round 5 at 74.77%, right before Test SR drops back to ~63%–68% in Rounds 6–9). Furthermore, the Sol run terminates early at Round 9 while Terra and DeepSeek run for 18–19 rounds, making cross-optimizer and cross-round comparisons inconsistent.

  3. High initial training saturation and missing hyperparameter specifications: The initial harness already achieves 88.75% success on the 160-task training split (leaving only 18 failed tasks initially), which severely limits the actual search space for the proposed failure-coverage algorithm and raises questions about the nearly 30-point train–test gap (88.75% vs. 59.06%). Moreover, key hyperparameters (KK, λ\lambda, softmax temperature, periodic full-evaluation frequency) are omitted from the paper, and the graph neighbor count contradicts between Appendix A (8 neighbors) and Appendix H Prompt 2 (at most 3 neighbors).

Questions for authors

See the weakness.

Reviewer_XDzSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCREENRSI, a harness-level recursive self-improvement (HRSI) framework for GUI agents that keeps the acting model and environment fixed while iteratively editing the harness. The key idea is to use a static semantic task-similarity graph to select diverse failure “seeds” and semantically close/distant “protected” successes, then perform branch-isolated repairs and hierarchical merges that must pass targeted regression checks before promotion. On MobileGym with Qwen3.8-27B fixed as the actor, SCREENRSI reports substantial test-set improvements (up to 74.77% SR, +15.71 points over the initial harness) and a reduction in false completions, with ablations showing that graph-guided selection outperforms random selection under the same round budget.

Critical strengths
  1. Introduces a principled, graph-guided allocation of scarce evaluation budget to both repair (diverse failure seeds) and preservation (semantically close and distant protected tasks), addressing a persistent challenge in harness evolution: local edits with global behavioral effects.
Critical weaknesses
  1. The paper cites several relevant threads (e.g., Meta-Harness, Self-Harness, HARNESSFIX, Harness-R1, GUI-HARVEST, HarnessOpt-Bench, SSR), but the empirical and conceptual comparison is light. In particular, connections to regression-aware validation (HARNESSFIX), evidence-driven multimodal diagnosis (GUI-HARVEST), and outcome-grounded online training of an editor (Harness-R1) are not explored experimentally.
  2. Insufficient detail on the graph construction rubric, validation of similarity scores, and selection hyperparameters (e.g., λ in the coverage-diversity objective, K seeds per round) in the main text; appendices are referenced but core sensitivity analyses are missing. Compute/budget description is vague: round budgets, per-round evaluation counts, and token costs for optimizer and actor are not fully specified; it is hard to assess cost-effectiveness or replicate the budget conditions in the ablation.
Questions for authors
  1. Non-selective results. Please report, in addition to the peak: (i) the final-round test SR, (ii) mean ± std over all evaluated rounds, and (iii) the result obtained when the round is chosen on a validation split carved out of the training set. How much of "+15.71" survives? Was the random baseline also reported at its own peak?
  2. Attribution. The backbone spread (74.77 vs 57.53, 17.24 points) is ~3× the graph-vs-random margin (5.73). How do you attribute the gain to the graph rather than to the optimizer? Please add an ablation with random seeds + the same protected-task budget so that "graph guidance" and "using preservation tasks" are separated.
  3. Does the graph capture what matters? Do graph neighbors co-fail empirically, and do accepted repairs propagate to neighbors significantly more than to non-neighbors? Which model built the graph, what is its self-consistency, and what does construction cost? Please also sweep λ, K, the temperature, and the number of protected tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.