- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose UI-MEM, a mechanism designed to mitigate gradient starvation within GRPO for mobile GUI agents. UI-MEM injects retrieved experience with additional guidance, thereby creating information differences. The authors show that UI-Mem outperforms zero-shot inference, vanilla-RL, and dense-reward training.
Critical strengths
(1) This method is well-motivated, and addresses a real issue, namely gradient starvation in GRPO, within GUI navigation, which is a highly relevant use case. Results indicate that the proposed approach works fairly well.
(2) The empirical investigation is well-scoped: Besides evaluations against base models on AndroidLab and AndroidWorld, the authors provide comparisons to other RL paradigms, ablation studies and a within-group variance analysis.
Critical weaknesses
(1) Main: The proposed method necessitates additional information, including a hierarchy of subtasks and additional guidance. It is not entirely clear how well this method generalizes beyond the presented benchmark. I would invite the authors to either demonstrate that their framework generalizes beyond individual GUI benchmarks (e.g., by clearly demonstrating that the hierarchies and template extend to other GUI interactions beyond the selected benchmarks; this would strengthen their contribution), or scope the following claims more carefully, which currently seem undersupported:
- „rigorously testing the agent’s reasoning and generalization capabilities“
- „demonstrating improved generalization ability“
- „with strong generalization to unseen applications.“
(2) The authors claim that varying the extent of guidance is required to avoid dependency overfitting: „However, simply prepending the same retrieved guidance to every rollout prompt suffers from the risk of dependency overfitting, where the agent learns to follow external instructions rather than internalizing transferable skills.“. While plausible, this claim should either be stated as a hypothesis, or demonstrated rigorously.
Questions for authors
- Could the authors please include a shortened version of their Limitations in the main body?
- Could the authors please discuss that the approach is only applicable in a single domain and requires substantial extra effort (creating a hierarchy, creating templates, etc.) as a Limitation, or explain why it does not?
- I appreciate that the authors commit to a full code-release. Would the authors please provide a link to an anonymized repository with their code?
Additional remarks:
- Experience replay dates back to Lin (1992), not Xu et al., 2025; Lu et al., 2025a.
- A more extensive discussion of previous work in experience replay would help to better motivate this method. This paper is about accelerating reinforcement learning by maximizing the learning potential of the provided samples, which mirrors traditional approaches in classic RL, such as Prioritized Experience Replay (ICLR 2016), Prioritized Level Replay (ICML 2021) or more recently, Uncertainty Prioritized Experience Replay (RLC 2025) or Reliability-adjusted Prioritized Experience Replay (ICLR 2026).
- Formulations like „guided exploration signals“ are, in my opinion, slightly misleading — this papers core contribution is about re-using past experiences, not exploration.
Minor remarks:
- Could the authors please clean up their References section for the camera-ready? Currently, there are some inconsistencies ("arXiv preprint arXiv:..." vs. "ArXiv, abs/..."; "Advances in Neural Information Processing Systems" vs "Advances in neural information processing systems")
- The authors core motivation rests on the reasoning that GRPO frequently samples groups in which all trajectories fail, yielding little signal. This failure mode should be demonstrated, not merely asserted, e.g., by forward-referencing to Figure 5b.
5 comments
Part 2/2: Main-body limitations (Q1), anonymized code repository (Q3), and responses to additional remarks
Q1: Including a shortened version of Limitations in the main body.
A: Thank you for this suggestion. In the revised paper, we have added a Limitations subsection in Section 5 (Main Body) summarizing: (1) the reliance on auxiliary LLM modules for training-time verification, automated subtask/essential-state construction, and experience extraction; (2) the scale and partial app-level overlap of the 256 seed queries on AndroidLab; (3) the current focus on a single domain (Android mobile GUIs), leaving extensions to web, desktop, and iOS environments to future work; and (4) the additional training effort for constructing task structure. Full details remain in Appendix K.
Q3: Anonymized repository link for code release.
A: We provide an anonymized repository, containing our online RL training framework (stratified group sampling, hierarchical memory retrieval and update, and two-stage reward verification), AndroidWorld evaluation suite (both policy-only and with frozen memory retrieval), auxiliary-model prompts, training/evaluation scripts, and sample task/subtask templates. Due to the file-size and storage limits of the anonymous hosting platform, the trained model checkpoints, full training dataset, and accumulated memory banks will be released upon acceptance.
Responses to Additional Remarks and Minor Remarks
Thank you for the valuable remarks, all of which we have incorporated into the revised manuscript:
-
Historical attribution and expanded discussion of Experience Replay (Lin, 1992; PER; PLR; UPER; RaPER): Thank you for this valuable suggestion. We have updated Section 1 to cite Lin (1992) for foundational Experience Replay alongside its recent adaptations to GUI agents (Xu et al., 2025; Lu et al., 2025). Furthermore, we have expanded Section 2 (Related Work) to discuss how UI-Mem relates to sample-prioritization paradigms in reinforcement learning, including Prioritized Experience Replay (Schaul et al., ICLR 2016), Prioritized Level Replay (Jiang et al., ICML 2021), Uncertainty Prioritized Experience Replay (Carrasco-Davis et al., RLC 2025), and Reliability-Adjusted Prioritized Experience Replay (Pleiss et al., ICLR 2026). Specifically, we highlight the conceptual connection: whereas prioritized replay and level replay maximize sample utility by re-weighting which historical transitions or environment seeds are re-sampled, UI-Mem maximizes within-group advantage utility in online group-based RL by structuring past experience into multi-level guidance to prevent zero-variance advantage collapse.
-
Revising formulations like "guided exploration signals": We agree with your observation that our core contribution lies in re-using past experience to construct informative within-group reward contrasts, rather than proposing an exploration algorithm. We have revised the phrasing in Sections 1, 3, and 4, replacing "converted into guided exploration signals" with "experience-guided rollout scaffolding" and "stratified experience reuse."
-
Cleaning up References formatting: We have thoroughly audited and standardized references in the revised paper to ensure consistent capitalization of venue names, unified arXiv formatting, and updated published conference entries where applicable.
-
Forward-referencing Figure 5(b) when motivating GRPO's all-failure collapse: Thank you for this helpful suggestion. We have added explicit forward references to Figure 5(b) and Section 4.4 in both Section 1 and Section 3.1 where we introduce the all-failure zero-variance collapse of standard GRPO.
Thank you for the detailed response. The changes you describe address my main concerns. However, the PDF currently on OpenReview does not appear to include these revisions yet. Could you please upload the revised version?
I am keeping my score. The method is well-executed and the gains are convincing. But it is evaluated in a single domain and relies on domain-specific task structure, such as subtask hierarchies, essential states, and templates. Rescoping the claims makes the paper more accurate, but it doesn’t broaden what the paper shows. I therefore see this as a solid, focused contribution to RL for mobile GUI agents rather than a broadly applicable advance.
If the revisions are made as described, I believe they improve the paper. I continue to support this paper's publication.
Part 1/2: Generalization scope & task-structure effort (W1/Q2) and dependency-overfitting analysis (W2)
Thank you for your insightful review and positive feedback on our paper. We greatly appreciate your recognition that our method is well-motivated, addresses gradient starvation in GRPO for GUI navigation, and is supported by a well-scoped empirical investigation. We have already incorporated all of your suggestions into the updated PDF and address each point below.
W1 & Q2: Generalization scope and the effort required to construct task structure.
A1: Thanks for raising this important point.
(a) Scoping the generalization claims carefully. Our current evidence supports transfer within Android GUI interaction, where our experiments evaluate two forms of transfer: (i) cross-benchmark transfer to AndroidWorld (116 tasks whose tasks and instruction sources are excluded from our training data); (ii) cross-application transfer to 5 held-out AndroidLab apps entirely unseen during training. We fully agree that this does not establish transfer to web, desktop, or arbitrary new GUI interaction domains, as noted in our Limitations (Appendix K, "Finally, our experiments are restricted to Android; scaling UI-Mem to broader task distributions and to other platforms such as web, desktop, and iOS remains an important direction for future work."). Following your suggestion, we have included a shortened limitation in the main part and revised all other statements so that our claims stay strictly within the evidence supported by our experiments:
(b) Discussing the extra effort of constructing hierarchies and templates. We also agree that UI-Mem requires extra setup and online effort during training. Task/subtask templates and essential-state definitions are constructed with LLM assistance from public training examples. The learned workflow, skill, and failure-pattern stores subsequently grow from online interaction. Automating annotation reduces manual work but does not eliminate data preparation and verification, and extending to other GUI domains requires domain-specific adaptation. We have added an explicit discussion in the Limitations (both Section 5 and Appendix K) stating both single-domain evaluation and additional training pipeline effort explicitly. At the same time, we clarify that constructing the hierarchy and templates does not require per-app manual engineering or test-time annotations.
W2: Rigorously demonstrating the
dependency overfittingclaim under uniform full guidance.A2: In the revised paper, we have (i) rephrased the statement in Section 3.2 as an explicit hypothesis motivating stratified sampling, and (ii) connected it directly to the experiment
w/o Stratified Sampling(Uniform Full Guidance) in Section 4.4 and Appendix J. As shown in the table below, when every rollout in every group receives uniform full guidance, the trained agent achieves 56.5% provided with external guidance at test time (w/ Retrieval), but collapses by 8.2 points to 48.3% when evaluated independently (Policy-Only). Moreover, its unguided performance (48.3%) is lower than Vanilla GRPO trained without any memory (53.0%). This demonstrates that uniform full guidance trains the model to rely on external instructions rather than internalizing standalone task-solving skills. We have added explicit cross-references between the hypothesis in Section 3.2 and this quantitative analysis in Section 4.4 / Appendix J.w/o Stratification)