acceptodds
Under review as a conference paper at ICLR 2027

Where to Learn and How Far to Roll Out: Efficient Adaptive Training for GUI Agents

Abstract

Trajectory-level GRPO is a common approach for training multi-step GUI agents, but repeatedly rolling out complete trajectories from the task root can be inefficient: task-level rewards provide coarse supervision over which intermediate decisions require improvement, while already-mastered prefixes are repeatedly re-executed before reaching difficult states. We introduce a two-stage adaptive training framework that jointly addresses where to learn and how far to roll out. We first construct reusable intermediate checkpoints from successful expert trajectories and select informative pivots using action disagreement, which measures uncertainty directly from sampled GUI actions. Stage I performs one-step RL at these pivots to improve uncertain local decisions. Stage II then progressively expands policy responsibility through adaptive Local, Suffix, and Root training according to task progress. Recoverable checkpoints enable repeated training from intermediate states without replaying the preceding trajectory. Across MobileGym and CUA-Gym-Hub, our method reduces environment interaction by 56-61% relative to Vanilla GRPO while improving task performance, demonstrating that jointly allocating training across states and rollout ranges can substantially improve GUI-agent RL efficiency.

Primary area:
reinforcement learning
Keywords:
GUI Agent, RL, Efficient Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LJVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes an adaptive reinforcement learning training framework for GUI agents. The method combines intermediate checkpoints with a two-phase training strategy that gradually expands from local updates to full task replay. Experiments on MobileGym and CUA-Gym-Hub demonstrate significant improvements in training efficiency and task performance.

Critical strengths

The motivation of the paper is clear, the methods are simple, intuitive, and reasonable, and it has certain practical application value.

The overall experiment is relatively complete, reflecting significant cost reduction and performance improvement.

Critical weaknesses

Intermediate state training and checkpoint recovery have been studied in works such as PivotRL and BPO. However, the paper lacks direct comparisons with these highly relevant baselines.

The core design of the method does not seem to take advantage of the uniqueness of the GUI environment and appears to be directly applicable to a broader range of agent tasks. However, the paper lacks discussion on this point and does not explain why it specifically focuses on the GUI environment.

Questions for authors

See Weaknesses

1 comment

We thank the reviewer for recognizing the practical value of our method and the improvements in training efficiency and task performance.

As the first author has recently left the institution and no longer has access to the original experimental infrastructure, we are currently unable to conduct additional experiments. We therefore address the concerns using existing evidence and clarify the scope of our work.

W1. Comparisons with PivotRL and BPO.

PivotRL. Its core approach focuses on selecting intermediate states using local reward variance and performing one-step RL, rather than full-trajectory training. We have already included a controlled comparison with this core mechanism in Section 4.2. Our Reward Variance baseline uses the same one-step RL procedure as Action Disagreement, differing only in pivot selection. On MobileGym, Action Disagreement achieves 48.83 vs. 45.70 Held-out Avg@4 under the same 8.2K budget; on CUA-Gym-Hub, the corresponding results are 52.08 vs. 47.50. This directly evaluates the effectiveness of our selection criterion relative to PivotRL-style reward-variance selection, although it is not a full reproduction of PivotRL.

BPO. While BPO also exploits intermediate states, its training strategy differs from ours. BPO selects high-entropy states from policy-generated trajectories and branches suffix rollouts for optimization. In contrast, our framework selects action-disagreement pivots from successful expert trajectories, first improves local decisions through one-step RL, and then progressively expands the rollout frontier using adaptive Local, Suffix, and Root training. Our focus is therefore not simply intermediate-state branching, but jointly determining where to learn and how far to roll out as training progresses.

W2. Why focus on GUI agents?

Our research was motivated by the practical application scenarios and research objectives of our institution, which primarily focus on GUI agents. We therefore designed and evaluated the framework specifically for multi-step GUI interaction tasks, where repeated environment interactions and intermediate-state recovery are important practical concerns.

Nevertheless, the underlying ideas of action-disagreement-based pivot selection and adaptive rollout allocation are not inherently restricted to GUI environments. We believe they could potentially extend to other interactive agent settings, which we consider a promising direction for future work.

Reviewer_BFnXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage adaptive training framework to improve the efficiency of multi-turn GUI agent RL. It first selects informative intermediate states from successful expert trajectories using action disagreement and performs one-step RL at these pivots. It then adaptively switches between local, suffix, and root-level training based on task success rates, leveraging reusable environment checkpoints to avoid redundant interactions. Experiments on MobileGym and CUA-Gym-Hub demonstrate reduced environment interaction costs and improved task performance compared with vanilla GRPO.

Critical strengths

S1. The paper addresses an important practical challenge in multi-turn GUI agent RL: reducing redundant environment interactions while alleviating the credit assignment problem. The proposed progressive training framework is intuitive and demonstrates promising improvements in interaction efficiency.

S2. The checkpointing mechanism enables efficient restoration of intermediate GUI states without replaying trajectory prefixes. This is a valuable engineering contribution with potential applicability beyond the proposed training framework.

Critical weaknesses

W1. Limited methodological novelty and insufficient ablations. Stage I largely follows PivotRL, while Stage II relies on heuristic training schedules. The authors should compare Action Disagreement against random pivot selection under matched training budgets, and Adaptive Hybrid against fixed-suffix training and predefined rollout curricula. These comparisons are necessary to establish whether the proposed adaptive mechanisms provide benefits beyond intermediate-state training itself.

W2. Potentially unreliable action matching. The action-matching criterion is overly coarse, particularly for text-entry actions, which are treated as equivalent regardless of their content. Since this criterion determines both pivot selection and local rewards, the authors should evaluate stricter parameter-aware matching and analyze its impact on training performance.

W3. Incomplete efficiency and generalization evaluation. The paper reports fewer environment interactions but lacks end-to-end training time comparisons accounting for expert trajectory generation, checkpointing, and pivot selection. The performance gains on MobileGym are also marginal, and multi-seed evaluations are needed to establish their reliability. Evaluation on broader benchmarks such as OSWorld or AndroidWorld would further strengthen the generalizability claim.

Questions for authors

Q1. Please address the concerns raised in the weaknesses.

Q2: Could the authors clarify the discrepancy in τsuffix\tau_{\mathrm{suffix}} between Section 4.1 (0.75) and Appendix A.3 (0.5), and confirm which value was used in the experiments?

1 comment

We thank the reviewer for recognizing the practical relevance of our work and the value of the checkpointing mechanism.

Before addressing the concerns, we would like to note that the first author has recently left the institution where the experiments were conducted and no longer has access to the original experimental infrastructure. As a result, we are currently unable to conduct additional experiments. Nevertheless, we will address the reviewer's concerns using the existing experimental evidence and revise the manuscript to clarify the methodological choices, evaluation scope, and limitations.

We respond to each concern below.

W1. Methodological novelty and ablations.

Our contribution lies in developing practical and effective training strategies, rather than introducing more complex metrics or schedules.

Stage I differs from PivotRL in its selection criterion: Reward Variance requires variation in local rewards, whereas Action Disagreement captures uncertainty directly from sampled GUI actions, even when all samples receive zero reward. Table 2 validates this distinction. On MobileGym, Action Disagreement achieves 48.83 vs. 45.70 Held-out Avg@4 under the same 8.2K budget. On CUA-Gym-Hub, the corresponding Train-selected results are 52.08 vs. 47.50.

Stage II is not a predefined curriculum. Training starts from Local, Suffix, or Root according to task success, and the suffix frontier advances only when the current segment becomes reliable. Both Base→Hybrid vs. Base→Vanilla and Action→Hybrid vs. Action→Vanilla demonstrate its effectiveness.

We agree that random-pivot and fixed-curriculum baselines would further isolate individual design choices. Nevertheless, our existing ablations provide evidence for the effectiveness of the proposed mechanisms relative to the evaluated alternatives.

W2. Action matching.

Our action-matching criterion is intentionally a lightweight approximation, not a perfect semantic evaluator. Its purpose is to provide inexpensive local training signals without additional models or complex verification.

We acknowledge that text-entry matching may overlook content differences. However, this criterion is used for pivot selection and local warm-up, while Stage II relies on task-level outcome rewards that can provide corrective feedback.

More complex matching is not necessarily better: LLM-based evaluation introduces additional cost and potential errors, while strict parameter matching may reject functionally equivalent actions. The practical value of a training signal lies in whether it improves downstream learning efficiently. Our results show that Stage-I warm-up improves held-out performance under both Hybrid and Vanilla training, supporting the usefulness of this approximation. We will clarify its limitations without claiming it is optimal.

W3. Efficiency and generalization.

Our primary goal is training efficiency, measured by environment rollout steps and policy samples entering optimization. These directly quantify two major training costs and are more hardware-independent than wall-clock time, particularly since our experiments used different GPU allocations and environment parallelism.

We acknowledge that one-time expert collection, pivot screening, and checkpoint preparation are excluded from these metrics. While complete wall-clock comparisons are unavailable, our checkpoint measurements show 71.6–81.1% lower restoration latency than replaying a 10-step prefix.

On MobileGym, the main benefit is efficiency rather than a large final accuracy gain. Action→Hybrid achieves 50.20 Held-out Avg@4 versus 49.61 for Action→Vanilla, using 40.8% fewer training samples and 50.2% fewer rollout steps.

While the final performance gain on MobileGym is modest, our method substantially improves training efficiency. On CUA-Gym-Hub, it achieves considerable improvements in both task performance and training efficiency. Together, these results provide consistent evidence of the effectiveness of our approach across two GUI benchmarks.

Q2. Suffix threshold.

The correct experimental value is τsuffix=0.75\tau_{\mathrm{suffix}}=0.75. The value 0.5 in Appendix A.3 is a typo and will be corrected.

Reviewer_9fzJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on GUI agents, where trajectory-level GRPO repeatedly rolls out complete trajectories from the task root, re-executing already-mastered prefixes and making credit assignment difficult across a multi-step trajectory. The authors propose to select informative pivots from successful expert trajectories using action disagreement and introduces a two-stage adaptive training framework. Stage I performs one-step RL at the selected pivots to improve uncertain local decisions. Stage II progressively expands policy responsibility through adaptive Local, Suffix, and Root training according to the policy’s performance. Experiments on MobileGym and CUA-Gym-Hub, including ablations of pivot selection and training configurations, demonstrate a strong balance between training cost and downstream task performance, improving training efficiency relative to Vanilla GRPO.

Critical strengths
  1. Well-motivated and targeted method. The paper addresses practical challenges in trajectory-level GRPO for GUI-agents : repeatedly re-executing already-mastered prefixes and obtaining useful learning signals from multi-step trajectories with task-level rewards. Pivot selection based on action-disagreement and the two-stage adaptive training framework are designed to address where to learn and how far to roll out. The method is consistent with the stated motivation and offers a potentially useful approach to improving training efficiency.
  2. Supportive ablation experiments. The experiments on MobileGym and CUA-Gym-Hub provide evidence for the effectiveness of action-disagreement pivot selection and the two-stage adaptive training framework. The comparison with SFT on the same expert data further supports the benefits of the RL procedure beyond supervised imitation. Together, these results show the method’s favorable balance between training cost and downstream task performance.
Critical weaknesses
  1. Requirements for checkpoint restoration. What requirements must an environment satisfy to support the checkpoint restoration method described in Section 3.1? Could it be applied to environments such as AndroidWorld and OSWorld? Clarifying these requirements would help establish the applicability of the proposed framework.
  2. Rationale for the task split. Section 4.1 motivates the custom MobileGym split by a substantial distribution gap between the official training and test sets. Could the authors further characterize this distribution gap, explain why a custom split is necessary, and clarify whether the gap reveals any limitations in the method’s generalization ability?
  3. Analysis of the Stage-I warm-up. The “Two-Stage Training” paragraph states that it examines whether Action-Disagreement warm-up further improves multi-step training, but compares Action→Hybrid with Action→Vanilla, both of which include the Stage-I warm-up. Would comparing Action→Hybrid with Base→Hybrid, or Action→Vanilla with Base→Vanilla, more directly assess the contribution of Stage I?
Questions for authors

See above.

1 comment

We thank the reviewer for the positive assessment of our method and experimental results. We appreciate the constructive questions and provide the following clarifications.

W1. Requirements and applicability of checkpoint restoration.

Our checkpoint mechanism primarily requires an environment that can be deployed and restored within a containerized runtime. In our implementation, we use Podman and CRIU to save and restore the executable environment state, allowing repeated rollouts from the same intermediate state without replaying preceding actions.

In general, containerized environments are compatible with this approach, provided their runtime state can be reliably restored. For Android environments, we have successfully supported several applications, although Chrome running inside an Android Virtual Device (AVD) presents additional restoration challenges. Similarly, containerized OSWorld environments can potentially use the same mechanism, subject to environment-specific validation.

We will clarify these requirements and practical limitations in Section 3.1 and Appendix A.1.

W2. Rationale for the MobileGym task split.

The main reason for constructing a custom MobileGym split is the substantial difficulty imbalance between the official training and test sets. Our base policy achieves approximately 89% success on the original training set, which consists predominantly of relatively easy tasks requiring only a few interaction steps. In contrast, the original test set contains substantially more challenging tasks, typically requiring longer interaction trajectories, and the policy achieves much lower success rates.

This imbalance makes the original training set unsuitable for studying efficient RL training, since most tasks are already solved reliably and provide limited room for further improvement. We therefore construct a more challenging training split from the official 256-task test set, dividing it into two disjoint subsets. After verifying expert executability, 83 tasks are retained for training, while the remaining 128 tasks are used exclusively for held-out evaluation.

This protocol allows us to study training efficiency on tasks that still require substantial improvement, while evaluating performance on disjoint, unseen tasks. We acknowledge that it does not establish generalization across the original train-test difficulty gap.

W3. Contribution of the Stage-I warm-up.

We agree that comparing Action→Hybrid with Action→Vanilla does not isolate Stage I, since both methods use the same warm-up initialization. Our experiments include two complementary sets of comparisons that examine the contributions of the two stages separately.

First, to evaluate Stage I, we compare Action→Hybrid with Base→Hybrid and Action→Vanilla with Base→Vanilla. On held-out MobileGym tasks, adding Stage I improves Avg@4 from 47.85 to 50.20 under Hybrid training and from 47.66 to 49.61 under Vanilla GRPO. On CUA-Gym-Hub, the corresponding improvements are from 55.83 to 57.92 and from 45.83 to 53.33, respectively. These results support the benefit of Action-Disagreement warm-up under both subsequent training procedures.

Second, to evaluate Stage II, we compare Action→Hybrid with Action→Vanilla, as well as Base→Hybrid with Base→Vanilla. In particular, the Action→Hybrid versus Action→Vanilla comparison is intended to verify that adaptive hybrid training remains more effective than Vanilla GRPO even when both methods start from the same Stage-I warm-up policy. Figure 4 shows that Hybrid achieves stronger performance with substantially fewer training samples and environment interactions.

We thank the reviewer again for the suggestions, which will help improve the clarity of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.