acceptodds
Under review as a conference paper at ICLR 2027

UI-Mem: Stratified Memory-Guided Online Reinforcement Learning for Mobile GUI Agents

Abstract

Online Reinforcement Learning (RL) offers a promising paradigm for enhancing mobile GUI agents through direct environment interaction. However, its effectiveness is severely hindered by long horizons and sparse rewards: under group-based methods such as GRPO, sampled rollout groups often collapse into uniform failure, yielding near-zero advantage and little relative learning signal. We propose UI-Mem, a framework that repurposes past experience as a training-time scaffold for online RL. Instead of merely replaying successful trajectories or scoring rollouts after collection, UI-Mem uses stratified memory-guided group sampling: within each GRPO group, retrieved experience is injected at different strengths to produce trajectories with full, partial, or no guidance. These contrasted rollouts create informative reward differences and train the unguided policy to internalize the external experience. UI-Mem instantiates this scaffold with a hierarchical experience memory, where workflows, subtask skills, and failure patterns are stored as parameterized templates and updated online from newly collected successes and failures. Experimental results show that UI-Mem substantially outperforms vanilla RL, dense-reward training, and static experience reuse across model scales, with strong generalization to unseen applications.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Online Reinforcement Learning, Mobile GUI Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wrjeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose UI-MEM, a mechanism designed to mitigate gradient starvation within GRPO for mobile GUI agents. UI-MEM injects retrieved experience with additional guidance, thereby creating information differences. The authors show that UI-Mem outperforms zero-shot inference, vanilla-RL, and dense-reward training.

Critical strengths

(1) This method is well-motivated, and addresses a real issue, namely gradient starvation in GRPO, within GUI navigation, which is a highly relevant use case. Results indicate that the proposed approach works fairly well.

(2) The empirical investigation is well-scoped: Besides evaluations against base models on AndroidLab and AndroidWorld, the authors provide comparisons to other RL paradigms, ablation studies and a within-group variance analysis.

Critical weaknesses

(1) Main: The proposed method necessitates additional information, including a hierarchy of subtasks and additional guidance. It is not entirely clear how well this method generalizes beyond the presented benchmark. I would invite the authors to either demonstrate that their framework generalizes beyond individual GUI benchmarks (e.g., by clearly demonstrating that the hierarchies and template extend to other GUI interactions beyond the selected benchmarks; this would strengthen their contribution), or scope the following claims more carefully, which currently seem undersupported:

  • „rigorously testing the agent’s reasoning and generalization capabilities“
  • „demonstrating improved generalization ability“
  • „with strong generalization to unseen applications.“

(2) The authors claim that varying the extent of guidance is required to avoid dependency overfitting: „However, simply prepending the same retrieved guidance to every rollout prompt suffers from the risk of dependency overfitting, where the agent learns to follow external instructions rather than internalizing transferable skills.“. While plausible, this claim should either be stated as a hypothesis, or demonstrated rigorously.

Questions for authors
  • Could the authors please include a shortened version of their Limitations in the main body?
  • Could the authors please discuss that the approach is only applicable in a single domain and requires substantial extra effort (creating a hierarchy, creating templates, etc.) as a Limitation, or explain why it does not?
  • I appreciate that the authors commit to a full code-release. Would the authors please provide a link to an anonymized repository with their code?

Additional remarks:

  • Experience replay dates back to Lin (1992), not Xu et al., 2025; Lu et al., 2025a.
  • A more extensive discussion of previous work in experience replay would help to better motivate this method. This paper is about accelerating reinforcement learning by maximizing the learning potential of the provided samples, which mirrors traditional approaches in classic RL, such as Prioritized Experience Replay (ICLR 2016), Prioritized Level Replay (ICML 2021) or more recently, Uncertainty Prioritized Experience Replay (RLC 2025) or Reliability-adjusted Prioritized Experience Replay (ICLR 2026).
  • Formulations like „guided exploration signals“ are, in my opinion, slightly misleading — this papers core contribution is about re-using past experiences, not exploration.

Minor remarks:

  • Could the authors please clean up their References section for the camera-ready? Currently, there are some inconsistencies ("arXiv preprint arXiv:..." vs. "ArXiv, abs/..."; "Advances in Neural Information Processing Systems" vs "Advances in neural information processing systems")
  • The authors core motivation rests on the reasoning that GRPO frequently samples groups in which all trajectories fail, yielding little signal. This failure mode should be demonstrated, not merely asserted, e.g., by forward-referencing to Figure 5b.
5 comments
AuthorsOpenReview ↗

Part 1/2: Generalization scope & task-structure effort (W1/Q2) and dependency-overfitting analysis (W2)

Thank you for your insightful review and positive feedback on our paper. We greatly appreciate your recognition that our method is well-motivated, addresses gradient starvation in GRPO for GUI navigation, and is supported by a well-scoped empirical investigation. We have already incorporated all of your suggestions into the updated PDF and address each point below.

W1 & Q2: Generalization scope and the effort required to construct task structure.

A1: Thanks for raising this important point.

(a) Scoping the generalization claims carefully. Our current evidence supports transfer within Android GUI interaction, where our experiments evaluate two forms of transfer: (i) cross-benchmark transfer to AndroidWorld (116 tasks whose tasks and instruction sources are excluded from our training data); (ii) cross-application transfer to 5 held-out AndroidLab apps entirely unseen during training. We fully agree that this does not establish transfer to web, desktop, or arbitrary new GUI interaction domains, as noted in our Limitations (Appendix K, "Finally, our experiments are restricted to Android; scaling UI-Mem to broader task distributions and to other platforms such as web, desktop, and iOS remains an important direction for future work."). Following your suggestion, we have included a shortened limitation in the main part and revised all other statements so that our claims stay strictly within the evidence supported by our experiments:

  • Abstract & Conclusion: "with strong generalization to unseen applications" →\to "with consistent transfer to unseen Android applications within the evaluated mobile benchmarks."
  • Section 4.2: "rigorously testing the agent’s reasoning and generalization capabilities" →\to "evaluating the agent's multi-step execution across diverse task parameterizations in Android apps."
  • Section 4.4: "demonstrating improved generalization ability" →\to "demonstrating improved transfer to held-out mobile applications."

(b) Discussing the extra effort of constructing hierarchies and templates. We also agree that UI-Mem requires extra setup and online effort during training. Task/subtask templates and essential-state definitions are constructed with LLM assistance from public training examples. The learned workflow, skill, and failure-pattern stores subsequently grow from online interaction. Automating annotation reduces manual work but does not eliminate data preparation and verification, and extending to other GUI domains requires domain-specific adaptation. We have added an explicit discussion in the Limitations (both Section 5 and Appendix K) stating both single-domain evaluation and additional training pipeline effort explicitly. At the same time, we clarify that constructing the hierarchy and templates does not require per-app manual engineering or test-time annotations.

W2: Rigorously demonstrating the dependency overfitting claim under uniform full guidance.

A2: In the revised paper, we have (i) rephrased the statement in Section 3.2 as an explicit hypothesis motivating stratified sampling, and (ii) connected it directly to the experiment w/o Stratified Sampling (Uniform Full Guidance) in Section 4.4 and Appendix J. As shown in the table below, when every rollout in every group receives uniform full guidance, the trained agent achieves 56.556.5% provided with external guidance at test time (w/ Retrieval), but collapses by 8.28.2 points to 48.348.3% when evaluated independently (Policy-Only). Moreover, its unguided performance (48.348.3%) is lower than Vanilla GRPO trained without any memory (53.053.0%). This demonstrates that uniform full guidance trains the model to rely on external instructions rather than internalizing standalone task-solving skills. We have added explicit cross-references between the hypothesis in Section 3.2 and this quantitative analysis in Section 4.4 / Appendix J.

MethodGuidance During RolloutsZero-Var. Group Ratio (Early / Mid / Late)AndroidWorld SR w/ Retrieval (%)AndroidWorld SR Policy-Only (w/o Retrieval) (%)Retrieval Drop (Δ\Delta)
Base Model (Qwen3-VL-8B)——52.647.6−5.0-5.0
Vanilla GRPONone0.69 / 0.55 / 0.5955.253.0−2.2-2.2
Uniform Full Guidance (w/o Stratification)100% Full0.41 / 0.52 / 0.4656.548.3−8.2-8.2
UI-Mem (Stratified Group Sampling)Mixed Guidance0.34 / 0.26 / 0.2271.166.8−4.3-4.3
AuthorsOpenReview ↗

Part 2/2: Main-body limitations (Q1), anonymized code repository (Q3), and responses to additional remarks

Q1: Including a shortened version of Limitations in the main body.

A: Thank you for this suggestion. In the revised paper, we have added a Limitations subsection in Section 5 (Main Body) summarizing: (1) the reliance on auxiliary LLM modules for training-time verification, automated subtask/essential-state construction, and experience extraction; (2) the scale and partial app-level overlap of the 256 seed queries on AndroidLab; (3) the current focus on a single domain (Android mobile GUIs), leaving extensions to web, desktop, and iOS environments to future work; and (4) the additional training effort for constructing task structure. Full details remain in Appendix K.

Q3: Anonymized repository link for code release.

A: We provide an anonymized repository, containing our online RL training framework (stratified group sampling, hierarchical memory retrieval and update, and two-stage reward verification), AndroidWorld evaluation suite (both policy-only and with frozen memory retrieval), auxiliary-model prompts, training/evaluation scripts, and sample task/subtask templates. Due to the file-size and storage limits of the anonymous hosting platform, the trained model checkpoints, full training dataset, and accumulated memory banks will be released upon acceptance.

Responses to Additional Remarks and Minor Remarks

Thank you for the valuable remarks, all of which we have incorporated into the revised manuscript:

  • Historical attribution and expanded discussion of Experience Replay (Lin, 1992; PER; PLR; UPER; RaPER): Thank you for this valuable suggestion. We have updated Section 1 to cite Lin (1992) for foundational Experience Replay alongside its recent adaptations to GUI agents (Xu et al., 2025; Lu et al., 2025). Furthermore, we have expanded Section 2 (Related Work) to discuss how UI-Mem relates to sample-prioritization paradigms in reinforcement learning, including Prioritized Experience Replay (Schaul et al., ICLR 2016), Prioritized Level Replay (Jiang et al., ICML 2021), Uncertainty Prioritized Experience Replay (Carrasco-Davis et al., RLC 2025), and Reliability-Adjusted Prioritized Experience Replay (Pleiss et al., ICLR 2026). Specifically, we highlight the conceptual connection: whereas prioritized replay and level replay maximize sample utility by re-weighting which historical transitions or environment seeds are re-sampled, UI-Mem maximizes within-group advantage utility in online group-based RL by structuring past experience into multi-level guidance to prevent zero-variance advantage collapse.

  • Revising formulations like "guided exploration signals": We agree with your observation that our core contribution lies in re-using past experience to construct informative within-group reward contrasts, rather than proposing an exploration algorithm. We have revised the phrasing in Sections 1, 3, and 4, replacing "converted into guided exploration signals" with "experience-guided rollout scaffolding" and "stratified experience reuse."

  • Cleaning up References formatting: We have thoroughly audited and standardized references in the revised paper to ensure consistent capitalization of venue names, unified arXiv formatting, and updated published conference entries where applicable.

  • Forward-referencing Figure 5(b) when motivating GRPO's all-failure collapse: Thank you for this helpful suggestion. We have added explicit forward references to Figure 5(b) and Section 4.4 in both Section 1 and Section 3.1 where we introduce the all-failure zero-variance collapse of standard GRPO.

Thank you for the detailed response. The changes you describe address my main concerns. However, the PDF currently on OpenReview does not appear to include these revisions yet. Could you please upload the revised version?

I am keeping my score. The method is well-executed and the gains are convincing. But it is evaluated in a single domain and relies on domain-specific task structure, such as subtask hierarchies, essential states, and templates. Rescoping the claims makes the paper more accurate, but it doesn’t broaden what the paper shows. I therefore see this as a solid, focused contribution to RL for mobile GUI agents rather than a broadly applicable advance.

If the revisions are made as described, I believe they improve the paper. I continue to support this paper's publication.

Reviewer_rBr4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UI-Mem mixes fully guided, partially guided, and unguided rollouts within each GRPO group. Guidance comes from a memory of reusable workflows, subtask skills, and failure patterns that evolves during training. The aim is to avoid uniform-failure groups and improve the policy even when memory is unavailable at inference. Experiments on AndroidWorld and AndroidLab show substantial gains, supported by multi-seed ablations, held-out-app evaluation, and tests of auxiliary-model replacements and noisy memory.

Overall assessment: The policy-only gains and extensive supporting controls justify acceptance. The main remaining issue is attribution: the evidence supports memory-guided training more strongly than the specific mixed-group credit-assignment explanation.

Critical strengths
  • The improvement is substantial and persists without inference-time memory. Policy-only UI-Mem reaches 66.8% on AndroidWorld versus 53.0% for vanilla GRPO and 57.3% with progress rewards (Table 3). Three-seed results confirm a large advantage, and performance improves on the 59 unseen-app AndroidLab tasks (Tables 9 and 11). These controls support genuine policy improvement rather than only better prompting at test time.
  • The supporting evaluation addresses several plausible alternative explanations. The appendices include multi-seed component ablations, an unguided-success bonus sweep, train-test overlap checks, smaller/open auxiliary models, and corrupted-memory tests. The progress-reward baseline shares the initial subtask annotations, and useful gains remain with the bonus removed (Appendix J).
Critical weaknesses
  • The key stratification ablation changes more than group construction. The "without stratification" variant gives every rollout full guidance (Appendix J). This removes unguided practice and the guidance curriculum as well as mixed-group normalization. Its drop therefore does not isolate the claimed benefit of contrasting guidance levels within a group. A control with the same guidance proportions, curriculum, rewards, and memory, but homogeneous groups or within-guidance normalization, is needed. More reward variance can arise from different prompt difficulty; by itself, it does not show better credit assignment for the unguided policy.
Questions for authors
  1. Decisive experiment: Does mixed-group normalization outperform a guidance-budget-matched control that groups or normalizes each guidance level separately? This would distinguish the proposed optimization mechanism from ordinary training with varying amounts of guidance.
  2. Clarification: What exactly is averaged in Table 12? For Zoom, five binary-success tasks are reported as 80.0% +/- 6.7% over three evaluation runs. Please specify repetitions per task, the averaging unit, and whether these are standard deviations or standard errors. Also confirm that evaluation memory is frozen and never updated using test outcomes.
2 comments
AuthorsOpenReview ↗

Part 1/2: Controlled ablation isolating mixed-group normalization (W1/Q1)

Thank you for your comprehensive review and supportive feedback on our paper. We appreciate your recognition of the substantial improvement of our method without inference-time memory and the comprehensiveness of our supporting evaluation. Below are our detailed responses to the points you raised.

W1 & Q1: Isolating mixed-group normalization against a guidance-budget-matched control.

A1: Thank you for pointing out this confounding factor. To cleanly isolate the effect of mixed-group normalization from ordinary training with varying amounts of guidance, we evaluate Homogeneous Groups with Matched Guidance and Curriculum on AndroidWorld. In this setting, instead of mixing guidance levels within a group, all G=4G=4 rollouts in a given group receive the same guidance level z∈{full,partial,none}z \in \{\mathrm{full}, \mathrm{partial}, \mathrm{none}\} according to the current per-task curriculum probabilities (λfull,λpartial,λnone)(\lambda_{\text{full}}, \lambda_{\text{partial}}, \lambda_{\text{none}}). Across groups in each training iteration, the guidance levels are adjusted according to the exact same task-level dynamic curriculum proportions λz(St)\lambda_z(S_t) as UI-Mem. Advantages are normalized within each homogeneous group of size G=4G=4 (which equivalently normalizes each guidance level separately while preserving an effective group size of G=4G=4). This guidance-budget-matched control matches UI-Mem in overall guidance proportions, unguided practice, curriculum withdrawal, rewards, memory, and rollout budget, removing only mixed-group normalization.

The comparison results are shown below. Moving from Uniform Full Guidance (48.5548.55%) to Homogeneous Groups with Matched Guidance and Curriculum (58.7658.76%) improves policy-only performance because the policy is now exposed to unguided practice and curriculum withdrawal. However, it still falls behind UI-Mem (66.9766.97%, as well as 63.9763.97% at α=0\alpha=0) by a clear margin under the exact same guidance budget, curriculum, rewards, and memory. This is because even though the overall guidance distribution is identical to UI-Mem, unguided rollouts are only normalized against other unguided rollouts (which frequently all fail on hard tasks, yielding zero variance and zero gradient), while full-guidance rollouts are only normalized against other full-guidance rollouts (which increasingly all succeed, again collapsing variance). By contrast, UI-Mem's mixed-group normalization places guided positive anchors and unguided attempts inside the same group, showcasing a lower zero-variance group ratio and directly improving credit assignment for the unguided policy by ensuring that unguided failures receive negative advantages and partial unguided progress receives positive advantages. We have added this controlled comparison to Section 4.4 and Appendix J of the revised paper.

MethodEffective Group SizeGuidance Proportions & CurriculumNormalization ScopeZero-Var. Group Ratio (Early / Mid / Late)AndroidWorld SR (Policy-Only, %)
Vanilla GRPO4100% noneWithin group0.69 / 0.55 / 0.5949.57 ± 2.46
No-Guidance Only (GRPO + Progress Reward)4100% noneWithin group0.54 / 0.43 / 0.4157.17 ± 1.23
Uniform Full Guidance (w/o Stratification, Appendix J)4100% fullWithin group0.41 / 0.52 / 0.4648.55 ± 1.97
Homogeneous Groups w/ Matched Guidance & Curriculum4Matched to UI-MemWithin guidance level (Homogeneous)0.46 / 0.42 / 0.3958.76 ± 1.08
UI-Mem (α=0\alpha=0, Mixed-Group Normalization)4Matched to UI-MemAcross mixed group0.35 / 0.28 / 0.2463.97 ± 0.54
UI-Mem (Default α=0.2\alpha=0.2, Mixed-Group Normalization)4Matched to UI-MemAcross mixed group0.34 / 0.26 / 0.2266.97 ± 0.54
AuthorsOpenReview ↗

Part 2/2: Held-out evaluation protocol and frozen memory confirmation (Q2)

Q2: Clarification of the Table 12 evaluation protocol and frozen memory during evaluation.

A2: We apologize for the brevity of the caption in Table 12 and clarify each point below:

(a) Repetitions per task, averaging unit, and standard deviation in Table 12. The results in Table 12 are aggregated over our 3 independently trained random-seed checkpoints (the same 3 training seeds as in Table 9), with 3 repeated evaluation runs per task for each checkpoint. Specifically, for an app aa with NaN_a binary-outcome tasks (e.g., Na=5N_a=5 on Zoom), evaluating each task 33 times under seed checkpoint r∈{1,2,3}r \in \{1, 2, 3\} yields 3Na3N_a binary trials (1515 on Zoom, 4545 on Bluecoins and Maps.me, 3636 on Cantook and Pi-Music, and 59×3=17759 \times 3 = 177 trials in total). Let crc_r denote the total number of trials with a successful outcome out of 3Na3N_a for checkpoint rr. The averaging unit is the checkpoint-level success rate SRa,r=cr/(3Na)\mathrm{SR}_{a,r} = c_r / (3N_a) (which is identical to averaging the three single-pass task scores, e.g., averaging 6060%, 8080%, 8080% to obtain 11/15=73.311/15 = 73.3% on Zoom). Table 12 then reports the mean ±\pm sample standard deviation (ddof=1\mathrm{ddof}=1, not standard error) across the 3 checkpoints' scores.

(b) Per-run success counts for Table 12. We provide the complete per-checkpoint trial counts (c1,c2,c3)/3Na(c_1, c_2, c_3) / 3N_a and corresponding percentages below, and have added these details to Appendix J (Table 12):

Held-out AppTasks (NaN_a)Trials / Seed (3Na3N_a)UI-Mem (Policy-Only) Counts (c1,c2,c3)/3Na(c_1, c_2, c_3) / 3N_a (%)Policy-Only SR (Mean ± Sample Std)UI-Mem (+Retrieval) Counts (c1,c2,c3)/3Na(c_1, c_2, c_3) / 3N_a (%)+Retrieval SR (Mean ± Sample Std)
Bluecoins1545(14,15,16)/45(14, 15, 16) / 45 (31.1,33.3,35.6)(31.1, 33.3, 35.6)33.3 ± 2.2%(22,25,25)/45(22, 25, 25) / 45 (48.9,55.6,55.6)(48.9, 55.6, 55.6)53.3 ± 3.8%
Cantook1236(17,17,20)/36(17, 17, 20) / 36 (47.2,47.2,55.6)(47.2, 47.2, 55.6)50.0 ± 4.8%(22,25,25)/36(22, 25, 25) / 36 (61.1,69.4,69.4)(61.1, 69.4, 69.4)66.7 ± 4.8%
Maps.me1545(16,16,13)/45(16, 16, 13) / 45 (35.6,35.6,28.9)(35.6, 35.6, 28.9)33.3 ± 3.8%(16,13,16)/45(16, 13, 16) / 45 (35.6,28.9,35.6)(35.6, 28.9, 35.6)33.3 ± 3.8%
Pi-Music1236(10,7,10)/36(10, 7, 10) / 36 (27.8,19.4,27.8)(27.8, 19.4, 27.8)25.0 ± 4.8%(13,13,10)/36(13, 13, 10) / 36 (36.1,36.1,27.8)(36.1, 36.1, 27.8)33.3 ± 4.8%
Zoom515(6,6,6)/15(6, 6, 6) / 15 (40.0,40.0,40.0)(40.0, 40.0, 40.0)40.0 ± 0.0%(11,12,13)/15(11, 12, 13) / 15 (73.3,80.0,86.7)(73.3, 80.0, 86.7)80.0 ± 6.7%
Task-weighted59177(63,61,65)/177(63, 61, 65) / 177 (35.6,34.5,36.7)(35.6, 34.5, 36.7)35.6 ± 1.2%(84,88,89)/177(84, 88, 89) / 177 (47.5,49.7,50.3)(47.5, 49.7, 50.3)49.2 ± 1.6%

(c) Confirmation that evaluation memory is frozen. Yes, the experience memory is strictly frozen during all evaluations and is never updated using test outcomes. As noted in Appendix I ("Compute Resources and Wall-Clock Overhead"), the auxiliary modules for reward verification, experience extraction, abstraction, and memory update are used exclusively during training; at inference time, UI-Mem (policy-only) uses no memory at all, and UI-Mem (with retrieval) performs read-only retrieval from the fixed memory bank accumulated during training. Test trajectories and test outcomes are never used to add entries or update memory statistics. We have revised Section 4.3 and Appendix J to make this clear.

Reviewer_X1LiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UI-Mem, a memory-guided online RL framework for mobile GUI agents. It organizes experience into workflows, subtask skills, and failure patterns, and mixes trajectories generated with full guidance, partial guidance, or no guidance within GRPO groups. Experiments on AndroidWorld and AndroidLab report improvements, and including gains without inference-time memory retrieval.

Critical strengths

Using past experience to guide new rollouts is a sensible way to address exploration difficulties in GUI training. Workflows, execution skills, and failure patterns provide guidance at different levels, while the evolving memory allows newly discovered experience to support subsequent training.

The policy-only results are encouraging. On AndroidWorld, UI-Mem-8B achieves 66.8%, compared with 53.0% for vanilla GRPO and 57.3% for GRPO with progress rewards. I also appreciate the multi-seed experiments, auxiliary-model replacements, noise evaluations, and overlap checks in the appendices. The improvement from 27.1% to 35.6% on 59 tasks from unseen applications provides additional evidence of transfer.

Critical weaknesses

W1: The main ablation does not isolate stratified grouping. Appendix J states that every rollout in the variant without stratified sampling receives full guidance. This simultaneously changes the guidance distribution, removes unguided training trajectories, and changes the within-group comparison structure. Therefore, the difference between 66.8% and 48.3% does not establish how much improvement comes specifically from stratified grouping. Some gains could come from unguided training coverage and guidance withdrawal. The α=0 result shows that the gains do not rely entirely on the additional unguided-success bonus, but does not resolve this confounding.

W2: Important methodological precedents are not discussed. G²RPO-A studies guided and unguided candidates within the same GRPO group and adaptively adjusts guidance length. Guide-GRPO introduces hints after all unguided rollouts fail and adjusts importance weights to target unguided behavior. Agent Workflow Memory extracts parameterized workflows and accumulates memory online. UI-Mem’s integration of evolving GUI memory with online RL may still be valuable, but its contribution relative to these methods needs clearer explanation and comparison.

W3: Key implementation details remain unclear. With G=4, the partial-guidance ratio of 0.05–0.15 always gives an initial floor allocation of zero. Its inclusion therefore depends on an unspecified rounding adjustment, despite the statement that each group preserves partial guidance. Algorithm 1 also does not explain its fallback when the workflow library is empty. Although task definitions are available beforehand, it is unclear how they support cold-start guidance and what initialization is retained in the ablation without the self-evolving loop.

W4: The evaluation protocol needs clarification. Table 12 reports 80.0±6.7% over three evaluation runs on five Zoom tasks with retrieval. If each task is evaluated once per run with a binary outcome, this mean and standard deviation cannot arise under either the usual sample or population definition. Multiple instances or repeated attempts could explain the result, but the paper needs to specify the repetition and aggregation procedure, including how evaluation runs relate to training seeds.

Questions for authors

Q1: Could you address W1 with comparisons that match the overall guidance proportions, curriculum, rewards, and rollout budget? Comparisons between normalization schemes should also match effective group sizes.

Q2: Please clarify the allocation and initialization choices in W3, and provide the per-run counts and aggregation procedure for W4. Is memory frozen during evaluation?

Q3: Please clarify the implemented UCB score, whether progress rewards count subtasks or essential states, and whether and where policy-update time is included in Table 8.

2 comments
AuthorsOpenReview ↗

Part 1/2: Stratified grouping ablation (W1/Q1) and comparison with prior works (W2)

Thank you for your careful review and constructive feedback. We appreciate your recognition of our motivation, policy-only improvements, and supporting evaluations. Below are our responses to your concerns.

W1 & Q1: Isolating the effect of stratified grouping.

A1: To isolate stratified grouping while matching all other factors, we evaluate Homogeneous Groups with Matched Guidance and Curriculum on AndroidWorld. Instead of mixing guidance levels within a group, all G=4G=4 rollouts in a group receive the same guidance level z∈{full,partial,none}z \in \{\text{full}, \text{partial}, \text{none}\} according to the exact same task-level curriculum proportions λz(St)\lambda_z(S_t) as UI-Mem, with advantages normalized within each homogeneous group (G=4G=4). This matches UI-Mem in guidance proportions, unguided coverage, curriculum, rewards, rollout budget, and effective group size, removing only mixed-group contrast.

As shown below, moving from Uniform Full Guidance (48.5548.55%) to Homogeneous Groups w/ Matched Guidance (58.7658.76%) improves policy-only SR by exposing the policy to unguided rollouts and curriculum withdrawal. Yet it still trails UI-Mem (66.9766.97%, and 63.9763.97% at α=0\alpha=0) under the exact same guidance budget, curriculum, and group size (G=4G=4). In homogeneous groups, unguided rollouts are normalized only against unguided rollouts (often all failing on hard tasks with zero variance), while full-guidance rollouts are normalized only against full-guidance rollouts (increasingly all succeeding, again collapsing variance). By contrast, UI-Mem places guided anchors and unguided attempts in the same group (G=4G=4), lowering zero-variance groups (0.34/0.26/0.22 vs. 0.46/0.42/0.39) so unguided failures get negative advantages and partial unguided progress gets positive advantages (added to Section 4.4 and Appendix J).

MethodGGGuidance & CurriculumNorm. ScopeZero-Var. Group Ratio (Early / Mid / Late)Policy-Only SR (%)
Vanilla GRPO4100% noneWithin group0.69 / 0.55 / 0.5949.57 ± 2.46
No-Guidance Only (GRPO + Progress Reward)4100% noneWithin group0.54 / 0.43 / 0.4157.17 ± 1.23
Uniform Full Guidance (w/o Stratification)4100% fullWithin group0.41 / 0.52 / 0.4648.55 ± 1.97
Homogeneous Groups w/ Matched Guidance4Matched to UI-MemWithin guidance0.46 / 0.42 / 0.3958.76 ± 1.08
UI-Mem (α=0\alpha=0, Stratified Mixed-Group)4Matched to UI-MemAcross mixed group0.35 / 0.28 / 0.2463.97 ± 0.54
UI-Mem (Default α=0.2\alpha=0.2, Stratified)4Matched to UI-MemAcross mixed group0.34 / 0.26 / 0.2266.97 ± 0.54

W2: Comparison with G2RPO-A\text{G}^2\text{RPO-A}, Guide-GRPO, and Agent Workflow Memory.

A2: Thank you for identifying these relevant works. We have added citations and discussions of all three in Sections 1, 2, and 3.2:

(a) G2RPO-A\text{G}^2\text{RPO-A} and Guide-GRPO:

  • Connection: Like UI-Mem, these two works incorporate guidance into GRPO rollouts to assist policy learning on hard queries.
  • Key distinctions:
    1. Source of guidance: G2RPO-A\text{G}^2\text{RPO-A} and Guide-GRPO target static single-turn reasoning where ground-truth solutions or hints exist per query. In multi-step GUI tasks, ground-truth trajectories are unavailable or costly, and static action prefixes do not fit stochastic UI transitions. UI-Mem instead builds guidance online from a self-evolving, parameterized template memory from the agent's own rollouts for cross-task/app transfer.
    2. Stratification mechanism: G2RPO-A\text{G}^2\text{RPO-A} varies ground-truth CoT prefixes, while Guide-GRPO injects hints post-hoc after all unguided rollouts fail. UI-Mem stratifies guidance within each group into three levels tailored to GUI interaction: full (workflow + skills + failure patterns), partial (workflow only), and none. This jointly trains the policy for planning, execution, and independent action.

(b) Agent Workflow Memory (AWM):

  • Connection: Both AWM and UI-Mem extract reusable, parameterized workflows from online trajectories for cross-task transfer.
  • Key distinctions:
    1. Training-time RL scaffold vs. inference-time prompting: AWM uses workflows only as in-context prompts for frozen models at test time. As shown in Table 3 (Inference-time Prompting: 52.652.6% vs. UI-Mem: 66.866.8% policy-only, 71.171.1% with retrieval), inference-time prompting underperforms when the base model cannot reliably execute retrieved plans. UI-Mem integrates evolving memory into the online RL loop via stratified sampling and curriculum dropout, internalizing skills into model weights.
    2. Hierarchical + corrective memory: Beyond AWM's positive workflows, UI-Mem separates workflows, subtask skills, and corrective failure patterns from failed rollouts.
AuthorsOpenReview ↗

Part 2/2: Guidance allocation & initialization (W3/Q2), evaluation protocol (W4/Q2), and additional clarifications (Q3)

W3 & Q2 (Part 1): Guidance allocation and initialization.

A3: We have updated Section 3.2, Algorithm 1 (Appendix F), and Section 4.4 to clarify each point:

(a) Group allocation rule at G=4G=4. Our allocation preserves at least one guided (full or partial) and one unguided trajectory per group by rounding full- and no-guidance counts and assigning the remainder to partial guidance: nfull=round(Gλfull),nnone=round(Gλnone),npartial=G−nfull−nnone.n_{\text{full}}=\text{round}(G\lambda_{\text{full}}),\quad n_{\text{none}}=\text{round}(G\lambda_{\text{none}}),\quad n_{\text{partial}}=G-n_{\text{full}}-n_{\text{none}}. Under our setting in Table 7 (G=4G=4, λfull∈[0.05,0.70]\lambda_{\text{full}} \in [0.05, 0.70], λnone∈[0.25,0.80]\lambda_{\text{none}} \in [0.25, 0.80]), (nfull,npartial,nnone)(n_{\text{full}}, n_{\text{partial}}, n_{\text{none}}) shifts from (3,0,1)(3, 0, 1) at St≤0.2S_t \le 0.2, through (2,1,1)(2, 1, 1) and (1,1,2)(1, 1, 2), to (0,1,3)(0, 1, 3) at St≥0.8S_t \ge 0.8 (updated in Section 3.2).

(b) Cold-start fallback in Algorithm 1. At initialization, predefined task templates, subtask definitions, and essential states are available, while learned workflow, skill, and failure-pattern stores start empty (Appendix B, Figure 6; Appendix J). In Algorithm 1, Phase 2 falls back to the ordered, instantiated subtask definitions when no learned workflow exists, and Phase 3 adds no tips or warnings when mid-level stores are empty. Thus, full- and partial-guidance rollouts initially share this template-derived plan while unguided rollouts receive none; subsequent verified rollouts update learned workflows, skills, and failure warnings (added to Algorithm 1).

(c) Initialization in w/o Self-Evolving Loop. In this ablation, the memory is frozen at the cold-start initialization described in (b) (the predefined task and subtask templates in Figure 6), with online updates (Algorithm 2) disabled.

W4 & Q2 (Part 2): Table 12 evaluation protocol and frozen memory.

A4: We have updated Section 4.3 and Appendix J (Table 12) to specify:

(a) Repetition, aggregation, and per-run counts. Table 12 aggregates over our 3 random-seed checkpoints (Table 9), with 3 repeated evaluation rollouts per task per checkpoint. For app aa with NaN_a binary tasks (Na=5N_a=5 on Zoom), 3 rollouts/task under seed r∈{1,2,3}r \in \{1, 2, 3\} yield 3Na3N_a binary trials (1515 on Zoom, 4545 on Bluecoins/Maps.me, 3636 on Cantook/Pi-Music, 177177 total). With crc_r successes out of 3Na3N_a, the checkpoint score is SRa,r=cr/(3Na)\text{SR}_{a,r} = c_r / (3N_a) (matching the mean of 3 single-pass scores, e.g., [60[60%, 8080%, 8080%]→11/15=73.3] \to 11/15 = 73.3% on Zoom). Table 12 reports mean ±\pm sample std (ddof=1\text{ddof}=1) across the 3 checkpoints:

AppNaN_a3Na3N_aPolicy-Only c1,c2,c3c_1, c_2, c_3 (%)Policy-Only SR+Retrieval c1,c2,c3c_1, c_2, c_3 (%)+Retrieval SR
Bluecoins154514, 15, 16 (31.1, 33.3, 35.6)33.3 ± 2.2%22, 25, 25 (48.9, 55.6, 55.6)53.3 ± 3.8%
Cantook123617, 17, 20 (47.2, 47.2, 55.6)50.0 ± 4.8%22, 25, 25 (61.1, 69.4, 69.4)66.7 ± 4.8%
Maps.me154516, 16, 13 (35.6, 35.6, 28.9)33.3 ± 3.8%16, 13, 16 (35.6, 28.9, 35.6)33.3 ± 3.8%
Pi-Music123610, 7, 10 (27.8, 19.4, 27.8)25.0 ± 4.8%13, 13, 10 (36.1, 36.1, 27.8)33.3 ± 4.8%
Zoom5156, 6, 6 (40.0, 40.0, 40.0)40.0 ± 0.0%11, 12, 13 (73.3, 80.0, 86.7)80.0 ± 6.7%
Task-weighted5917763, 61, 65 (35.6, 34.5, 36.7)35.6 ± 1.2%84, 88, 89 (47.5, 49.7, 50.3)49.2 ± 1.6%

(b) Frozen memory during evaluation. Yes, memory is strictly frozen during all evaluations (Appendix I): policy-only UI-Mem uses no memory, and UI-Mem (with retrieval) performs read-only retrieval without adding entries or updating statistics.

Q3: UCB score, progress rewards, and policy-update time.

A: We have updated the revised paper to clarify each item:

(a) Implemented UCB score. Following Appendix F, the first term is candidate pp's empirical success rate (with +1+1 smoothing): S(p)=Nsucc(p)Nused(p)+1+λucbln⁡(∑p′∈PNused(p′)+1)Nused(p)+1,S(p) = \frac{N_{\text{succ}}(p)}{N_{\text{used}}(p) + 1} + \lambda_{\text{ucb}} \sqrt{\frac{\ln (\sum_{p' \in \mathcal{P}} N_{\text{used}}(p') + 1)}{N_{\text{used}}(p) + 1}}, where P\mathcal{P} is the matched candidate pool, Nsucc(p)N_{\text{succ}}(p) and Nused(p)N_{\text{used}}(p) are success and usage counts, and λucb=0.5\lambda_{\text{ucb}} = 0.5 (unified across Appendices E and F).

(b) Progress rewards: subtasks vs. essential states. Progress rewards count verified essential states, which serve as the verifiable milestone criteria for subtasks (Figure 6, Appendix B; aligned across Eq. 4 in Section 3.2 and Appendix C).

(c) Policy-update time in Table 8. Table 8 originally covered only rollout collection, verification, and memory evolution, omitting the GRPO policy update. The policy-update time for both vanilla GRPO and UI-Mem has been added to Table 8 in Appendix I.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.