- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents ScreenRSI, a framework for iteratively optimizing GUI agent harnesses while keeping the acting model and environment fixed. It uses a semantic task-similarity graph to select failure tasks for coverage and diversity, together with previously successful tasks for regression protection. An optimizer agent analyzes execution trajectories and modifies the harness, with repair validation and controlled merging producing the next revision. Experiments on MobileGym demonstrate improvements over the initial harness and gains from graph-guided task selection relative to random selection.
Critical strengths
- The research question is worthwhile. Treating the harness as an optimization target and exploring how execution feedback can drive iterative revisions to its code and prompts is a promising direction. The work offers a useful perspective on improving GUI agent capabilities with a fixed underlying model.
- The graph-guided task selection design is well motivated. Using behavioral similarity to balance failure coverage and diversity, while selecting both nearby and distant successful tasks to check for regressions, addresses the tension between local repair and capability preservation under a limited evaluation budget. The comparison with random selection provides initial empirical support for this design.
Critical weaknesses
- The direction of GUI agent harness self-improvement is appealing, but all experiments start from a single custom initial harness. The method is not evaluated on established GUI agent frameworks, such as Agent-S3 [1] or OS-Symphony [2], or on harnesses with different architectures. This limits the broader significance of the results: the gains may depend partly on the particular starting harness, and the evidence does not establish the method’s general applicability to mature GUI agent harnesses.
- All experiments are conducted on MobileGym. Given the paper’s broader focus on GUI agent harnesses, evaluation solely in a mobile environment is insufficient to establish applicability across environments. Additional experiments on OSWorld [3] would strengthen the evidence that ScreenRSI generalizes across platforms.
- Failure attribution and repair generalization lack empirical analysis. The appendix prompts distinguish failures originating from the harness, model behavior, environment, and evaluator, but the paper does not report their distribution, attribution accuracy, or repair outcomes by source. Since model behavior failures can also trigger harness modifications, the aggregate success-rate gains do not establish whether improvements primarily reflect transferable repairs to harness defects or prompt and control-flow changes that compensate for local model errors.
[1] Gonzalez-Pumariega G, Tu V, Lee C L, et al. The Unreasonable Effectiveness of Scaling Agents for Computer Use. 2025.
[2] Yang B, Jin K, Wu Z, et al. OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026: 22300–22330.
[3] Xie T, Zhang D, Chen J, et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Advances in Neural Information Processing Systems, 2024, 37: 52040–52094.
Questions for authors
- What is the provenance of the initial harness, and how does its functionality compare with established GUI agent frameworks? Could the authors apply ScreenRSI to at least one existing framework, or to an enhanced baseline with standard memory, recovery, and validation mechanisms, and report both the additional gains and the customization required for integration?
- Can the method transfer to desktop GUI environments such as OSWorld?
- In the actual failure analyses, what proportion of failures falls into each source category, and what are the corresponding repair acceptance rates? Were the attributions manually checked? Could the authors report repair success, regressions, and gains on unseen tasks by failure source, and examine whether the final modifications include compensatory rules tailored to particular task patterns?