acceptodds
Under review as a conference paper at ICLR 2027

Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps

Abstract

While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents consistently outperform the SFT agent. Notably, our approach demonstrates stronger out-of-distribution generalization to novel environments and achieves performance competitive with significantly larger frontier models on certain benchmarks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, PPO, Computer Use Agents, UI Navigation Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AUi5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an online RL framework for UI navigation on live iOS applications. The agent is decomposed into an RL-trained planning model and a frozen grounding model. It uses LLM-as-a-judge dense step-level rewards together with terminal rewards, and optimizes the planner with PPO and a proposed neighbor-conditioned critic method, GiGPPO. Experiments on UINavBench, iOSWorld, and AndroidWorld show improvements over the authors’ SFT baseline, with the strongest result being GiGPPO on UINavBench.

Critical strengths
  1. To be honest, GUI agents in the iOS environment are fascinating and make for compelling reading. Currently, the vast majority of GUI-related work centers on Android and Linux—platforms that differ significantly from iOS at the data level. Furthermore, the scaling of virtual environments on iOS is an interesting topic in its own right.
Critical weaknesses
  1. There are substantial flaws in the experimental setup, baseline selection, and evaluation. If the primary goal of this paper is to validate an online reinforcement learning (RL) framework for GUI agents, rather than claiming the creation of a new state-of-the-art (SOTA) model, then the current experimental setup appears suboptimal. A more compelling design would utilize standard backbone models, such as established general-purpose models (e.g., Qwen2.5/3/3.5/3.6) or models specifically designed for GUIs (e.g., GUI-Owl, MAI-UI, UI-Venus, UI-TARS), and then evaluate performance gains in iOS environments following training within the proposed framework, alongside out-of-distribution (OOD) transfer capabilities in Android environments.

In contrast, the current system combines a Qwen3-VL-8B-Instruct planner, an internal 3B-parameter grounding model (described as an "internal 3B dense pre-trained model fine-tuned on a large-scale human-annotated dataset"), and Gemini-based components for swipe actions and reward determination. This setup is somewhat atypical, making it difficult to discern whether observed performance improvements stem from the RL framework itself or from the choice of specific internal or proprietary components; this also undermines the comparability and reproducibility of the results. Of course, if your primary objective is to showcase or establish a strong model baseline for the iOS domain, the current results lack competitiveness; the model underperforms compared to others on iOSWorld and UINavBench (the two benchmarks you prioritize most), let alone on other OOD benchmarks.

Furthermore, the results reported in the paper for Gemini and Qwen3.8-27B on AndroidWorld are difficult to reconcile with previously published data. For instance, the official score for Qwen3.8-27B is as high as 81.9, and UI-Venus achieved a score of 70.3 even when using the less powerful Qwen3.6-27B; in contrast, the scores reported in this paper range only from 49 to 60, raising questions regarding the reliability of the evaluation protocol and the comparative results on AndroidWorld. In my own experience, while achieving a score of over 80 on AndroidWorld with Gemini and Qwen3.8-27B might require some prompt and harness adaptation, reaching a score of over 70 is easily achievable.

  1. Method and Writing At this point, I think I understand why the authors opted for such a complex system design—it resembles a full-duplex, live interaction model (Planner + Actor). However, this raises a question regarding the paper's motivation: what is the rationale for employing this type of full-duplex, live interaction? Is it driven by specific real-world use cases, or is the goal to create a more seamless interaction between the user and the agent system? Furthermore, demonstrating the advantages of this approach requires supporting data (such as the system's average execution time, the latency distribution across the full action pipeline, and a comparison with the latency of end-to-end models) .
Questions for authors

Q1: In the context of online RL, how much time overhead is incurred by the step-level "Dense judge" and the trace-level "Terminal judge," respectively? How is the pipeline for this component designed?

Q2: In online RL, are rollout and training synchronous or asynchronous? Do they share the GPU, use time-division multiplexing, or operate on separate, independent GPUs? Is there a model offloading phase? If so, when does it occur, and what are the approximate synchronization and offloading overheads?

Reviewer_nPGnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online reinforcement learning for UI navigation on live virtual iOS devices. An SFT-initialized Qwen3-VL-8B planner produces step plans and remaining plans, while a separate grounding model executes actions. Gemini judges provide dense progress feedback and terminal outcomes, and a simulated user answers clarification requests. The planner is trained with PPO using discounted dense rewards and an undiscounted terminal reward. GiGPPO augments the critic with a perceptually similar state and its return from a sibling rollout. The infrastructure supports 512 simultaneous trajectories. Table 1 reports improvements over SFT on several settings, including AndroidWorld after iOS-only RL updates; the strongest results vary across PPO variants and benchmarks.

Critical strengths

A concrete online learning system for live iOS interaction

Sections 3.2–3.4 describe separate planning and grounding models, device control over VNC, snapshot restoration in roughly 10 seconds, and 512 concurrent device–agent loops (pp. 4–5, lines 199–229 and 247–258). Figure 1 connects SFT initialization, rollout collection, reward judges, and policy updates. Training through actual app interaction is a substantial engineering contribution. Snapshot reuse and independent device instances address practical obstacles to collecting fresh trajectories at scale, and the architecture concentrates RL on planning.

Thoughtful treatment of intermediate feedback and operational failures

Sections 3.4–3.5 specify progress, overcompletion, clarification, and terminal labels; distinguish dense from terminal discounting; and mask infrastructure failures from actor updates, critic targets, neighbor selection, and advantage normalization (pp. 6–7, lines 275–290 and 305–370). The critic also receives a dedicated warm-up before joint optimization (pp. 7–8, lines 373–384). These details identify real failure modes of online agent training. In particular, separating device failures from policy failures and explicitly handling actions after completion make the learning setup more credible and useful to practitioners.

Relevant ablations and evidence of transfer after iOS RL

Table 1 compares a common SFT starting point, terminal-only PPO, dense PPO, and GiGPPO on two iOS benchmarks and AndroidWorld (p. 8, lines 414–428). All three RL checkpoints improve on the AndroidWorld SFT rate of 4.7%, with standard PPO reaching 11.2%. Section 4.3 specifies common per-benchmark grounding configurations for the planner comparisons (p. 9, lines 432–437). The terminal-only control directly tests whether dense feedback helps, and the Android evaluation tests whether iOS RL updates can transfer beyond their interaction environment. This is useful evidence, although Android is already represented in the shared SFT data and the improvements vary by method and benchmark.

Critical weaknesses

The main results do not yet isolate a reliable GiGPPO or dense-reward advantage

stabilizes the critic

Source: Section 5, Conclusion; supporting results in Table 1 and Section 4.3; PDF p. 9; manuscript lines 479–480.

Concern. The claim of critic stabilization is not accompanied by a critic-error, explained-variance, or seed-variance measurement in the main body. Table 1 shows GiGPPO improving over standard PPO on UINavBench, but falling below it on both iOSWorld and AndroidWorld. Dense PPO improves over terminal-only PPO on two benchmarks, as the authors correctly state, but is lower on UINavBench (14.7% versus 17.6%). Its checkpoint also uses 70 updates rather than 75. The GiGPPO margin over Qwen3.8-27B on UINavBench is only 2.0 percentage points. With no paired uncertainty or repeated-run results in the main table, these findings support promising, setting-dependent improvements rather than an established general advantage or demonstrated critic stabilization.

Suggested clarification or experiment. Please compare standard PPO and GiGPPO with matched interaction and update budgets, report paired task outcomes and uncertainty from repeated evaluations, and include a small diagnostic of held-out critic prediction error or training variance. A neighbor ablation using no neighbor and a randomly selected sibling state would help establish whether perceptual matching contributes beyond additional critic inputs. Please qualify the conclusions by benchmark and method.

Discounting limits dense credit, but does not by itself establish prevention of delayed completion

prevent reward hacking

Source: Section 5, Conclusion; return and rewards in Section 3.5; PDF p. 9; manuscript lines 478–479.

Concern. The asymmetric return in Section 3.5 is sensible as a way to keep dense credit local while preserving terminal credit. However, for the same successful terminal outcome, a sequence of n positively judged dense steps has return 2 + sum_{j=0}^{n-1}(0.5)^j at its beginning, which still increases with n. Thus discounting bounds the incentive to accumulate dense rewards; it does not itself impose a preference for a shorter successful path. The explicit penalties for overcompletion and unproductive behavior are helpful, but their effectiveness also depends on judge accuracy. Section 3.5 reports an observed reward-hacking run with gamma = 1, which motivates the change. A matched comparison quantifying trajectory length and unnecessary actions would help establish how effectively gamma = 0.5 addresses the behavior.

Suggested clarification or experiment. Please distinguish the bounded-credit property from a guarantee against delayed completion. A matched-budget comparison of dense discount factors, including 1 and 0.5, should report verified completion, steps per successful task, repeated or unnecessary actions, and overcompletion rates. If efficiency is an intended objective, discuss or test a small step cost or an explicitly time-sensitive terminal objective.

The reliability of reward and outcome judgments needs an independent check

efficient and robust

Source: Section 3.4, Online Reinforcement Learning Setup, Reward Design; PDF p. 6; manuscript lines 270–274.

Concern. Dense judgments use the planner reasoning, action history, prior judge verdicts, and screenshots; the terminal judge also receives prior dense verdicts (p. 6, lines 275–290). This is a practical design, but correlated mistakes can propagate from local feedback into the terminal reward. Labels for genuine progress, unnecessary clarification, and unavoidable external barriers require interpretation. The two iOS benchmarks also use VLM-based outcome checks (p. 8, lines 390–397), while the main text provides no independent agreement or error analysis. An independent audit would help calibrate the learned rewards and determine how much confidence to place in the small reported success differences.

Suggested clarification or experiment. Please audit a blinded, stratified sample with human labels or app-state checks where feasible. Report agreement and false-positive/false-negative rates separately for progress, terminal success, overcompletion, and blocked-completion cases. Recomputing terminal judgments without the earlier dense verdicts on this sample would also measure how much those verdicts influence the final label. Describe benchmark-judge identity and independence from the training judges.

The complete system’s size, efficiency, and reproducibility are insufficiently specified

our 3B RL agents

Source: Section 4.3, Results; PDF p. 9; manuscript lines 449–452.

Concern. Section 3.3 initializes the RL-trained planner from Qwen3-VL-8B-Instruct (p. 5, lines 233–244), whereas Section 4.2 assigns 3B to the grounding model (p. 8, lines 403–410). The quoted comparison therefore appears to confuse planner and grounder size. Moreover, inference uses Gemini for swipes on UINavBench and for all grounding on iOSWorld and AndroidWorld, with a Gemini simulated user available during evaluation. This is relevant to the motivation of lower inference cost, even though grounding is held common within each benchmark. The main body supplies no complete per-task cost or latency accounting. It also relies on two in-house SFT datasets and an internal grounder without main-text dataset sizes, split manifests, or an accessible reproduction path.

Suggested clarification or experiment. Please correct the component sizes and give a compact per-benchmark component table. Report task latency, total model/API calls, and cost for the planner, grounder, and simulated user, with training judge costs separately. For reproducibility, provide dataset and task counts, app/task split definitions, overlap checks against evaluation tasks, and a release plan or a documented configuration using available grounding components.

Questions for authors
  1. At matched training budgets, how consistent are the dense-PPO and GiGPPO gains across seeds or repeated task evaluations, and does GiGPPO measurably reduce critic prediction error or training variance?

  2. Does gamma_dense = 0.5 reduce unnecessary actions and steps per independently verified success relative to gamma_dense = 1, beyond the effect of the explicit overcompletion and loop judgments?

  3. What is the independent agreement rate of the dense, terminal, and benchmark judges, especially for blocked-completion cases and when terminal judgments do not receive earlier dense verdicts?

  4. Is the RL planner 8B or 3B, what is the total inference cost including grounding and simulated-user calls, and what data/configuration can an independent group use to reproduce the result?

Reviewer_eBjvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a GUI planner through online interaction with live iOS applications, using a separate grounding model. It combines dense and terminal judge rewards, discounts only the dense component, and introduces GiGPPO, a critic conditioned on visually similar states from other rollouts. A virtual-device system supports large-scale collection. Evaluation on UINavBench, iOSWorld, and AndroidWorld shows some improvements over supervised training, but does not establish consistent benefits from the proposed algorithmic components.

Overall assessment: The infrastructure is promising, but mixed results and confounded ablations leave the main algorithmic and efficiency claims insufficiently supported.

Critical strengths
  • The live-iOS training infrastructure is a useful systems contribution. Snapshot-based resets and support for 512 concurrent trajectories address a difficult practical bottleneck. Keeping the grounder fixed during RL also makes planner adaptation easier to study.
  • The evaluation includes meaningful transfer and controls. The paper compares SFT, terminal-only PPO, dense PPO, and GiGPPO, and evaluates on Android as well as iOS. AndroidWorld improves from 4.7% for SFT to 11.2% for dense PPO, providing evidence of some transferable planning benefit (Table 1).
Critical weaknesses
  • The results are too mixed to support the headline claims. Dense PPO is worse than terminal-only PPO on UINavBench, 14.7% versus 17.6%. GiGPPO improves that benchmark but trails standard PPO on both others. Its advantage over Qwen3.8-27B is only 2.0 points on UINavBench. Appendix A.1 reports five evaluation samples per task, but there are no training-seed results or paired uncertainty estimates. These results do not yet establish consistent superiority or a reliable GiGPPO benefit.
  • The new mechanisms are not isolated by the ablations. The failed undiscounted run in Appendix A.2 uses GRPO, different reward magnitudes, and no negative penalties; the proposed setup also changes the optimizer and penalty scheme. It therefore cannot attribute reduced reward hacking specifically to asymmetric discounting. Similarly, the claim that neighbor conditioning stabilizes the critic lacks direct value-error or gradient-variance measurements and a control for simply adding critic capacity.
Questions for authors
  1. Clarification: Please provide training-seed variation, paired task-level intervals, and the checkpoint-selection rule for Table 1. Are the reported gains stable at matched rollout budgets?
  2. Decisive experiment: With PPO and the reward scheme fixed, what changes when only dense discounting or neighbor information is removed? A shuffled-neighbor or equally sized critic control would help isolate GiGPPO.
  3. Clarification: Is the planner 8B, as stated in Sec. 3.3 and Appendix A.1, or 3B, as stated in Sec. 4.3? What is the full inference cost, including Gemini grounding and simulated-user calls, relative to the frontier baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.