acceptodds
Under review as a conference paper at ICLR 2027

Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization

Abstract

Recent advances in Multimodal Large Language Models (MLLM) have substantially driven the progress of autonomous agents for Graphical User Interface (GUI). Nevertheless, in real-world applications, GUI agents often face non-stationary environments, leading to high computational costs for data curation and policy optimization. In this paper, we introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization. The former one aims to optimize a Q-model that can generate step-wise values to evaluate the contribution of a given action to task completion. The latter one takes step-wise samples from the state-action trajectory as inputs, and optimizes the policy via reinforcement learning with our agentic-Q. Our framework enjoys two favorable advantages: (i) all trajectories are produced by the policy itself, so that data collection costs are manageable; (ii) the policy update is decoupled from the environment, ensuring stable and efficient optimization. Empirical evaluations show that our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on navigation and grounding benchmarks and even surpassing contenders with larger scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, GUI Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TzuLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenges of training GUI agents in real-world environments, specifically the high cost of interaction and the issue of sparse rewards—where feedback is provided only at the end of a task (terminal rewards). The proposed method begins by collecting trajectories using an initial SFT (Supervised Fine-Tuning) policy and training an "agentic-Q" model based on terminal success or failure. This Q-model is then used to score actions sampled by the policy, facilitating single-step policy optimization. This approach eliminates the need for further real-world environment interactions during the reinforcement learning (RL) update phase. The authors report performance improvements over the baseline SFT policy on the WebVoyager and Online-Mind2Web benchmarks.

Critical strengths
  1. The research motivation is clear. Reducing environment calls during policy training by leveraging a Q-model is a practical and valuable strategy for lowering the training costs of GUI agents.

  2. The paper is well-written, the methodological diagrams are intuitive, and the overall training pipeline is easy to follow.

Critical weaknesses
  1. The Q-model is trained using trajectories collected by the initial SFT policy and terminal rewards; consequently, its value estimates are heavily dependent on the subsequent performance of that SFT policy. For instance, an action that is actually conducive to task completion might receive a low score from the Q-model if the SFT policy fails to follow up on it effectively. Furthermore, §3.4 does not clarify whether the Q-model is updated during the policy training phase. If it remains static, there is a risk that estimation errors for actions sampled by the evolving policy will increase. The authors should provide empirical validation—for example, by comparing a static Q-model against a setup where trajectories are periodically re-collected and the Q-model is updated—while also reporting the associated additional interaction costs.

  2. Digi-Q [1] has already implemented offline Q-guided policy training for GUIs and has compared Monte Carlo returns with TD learning. Therefore, using terminal returns to train a Q-function is not, in itself, a sufficient innovation. The paper should clarify what additional benefits its Q-estimation method and step-wise policy optimization provide over existing Q-guided approaches.

  3. The manuscript does not cite or discuss Digi-Q. Given the similarity in the overall approach, I suggest adding a citation and explicitly detailing the methodological differences.

References

[1] Hao Bai et al. Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents. ICLR, 2025.

Questions for authors
  1. Does the Q-function remain fixed during SWPO? If it is updated, please explain the update mechanism and whether it requires additional environment interactions.

  2. Have you verified whether, at different training stages, the Q-function’s ranking of candidate actions for a given state aligns with the actual outcomes of those actions when executed?

  3. Compared to simply sampling actions using an SFT-trained policy and then using the Q-function to select the highest-scoring action, how much additional gain does the single-step policy optimization (SWPO) provide?

Reviewer_w38xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework for training GUI agents with agentic-Q estimation and step-wise policy optimization. The main motivation is that online GUI environments are expensive and unstable, making trajectory-level RL difficult to scale.

The authors first train an Ovis2.5-9B model with SFT. They then collect self-generated trajectories and train a Q-model to predict final task success from individual state-action steps. This Q-model is used to provide rewards for step-wise policy optimization, together with reward filtering and adaptive group weighting.

The method achieves 78.79% on WebVoyager and 64.00% on Online-Mind2Web using S-GRPO, improving over the SFT model and trajectory-level GRPO.

Critical strengths

I like the motivation of this work. Online GUI environments can be slow and unstable, so separating policy optimization from environment execution is a useful direction. Using a Q-model to provide step-level feedback also makes the training more flexible than only using the final trajectory reward.

The experiments show quite good improvements. In particular, the comparison between trajectory-level GRPO and S-GRPO is useful, since they start from the same SFT model. The results on Online-Mind2Web also suggest that the improvement is not only limited to the training websites.

Critical weaknesses
  1. My biggest concern is the reliability of agentic-Q. The Q-model is trained by assigning the final trajectory result to all intermediate steps. This estimates success under the data collection policy, but it does not directly tell whether one individual action is good or bad. More importantly, after the policy changes during SWPO, the generated actions can be different from the original Q-model training distribution. I think the paper needs more evidence that Q-model rewards remain reliable under this distribution shift.

  2. The claimed training efficiency is not sufficiently quantified. The method still needs 16k environment trajectories to train agentic-Q, and also needs expert data for the initial SFT. It would be helpful to compare the complete environment interactions, GPU hours, and training wall-clock time against online GRPO. Also, since training uses tasks from WebVoyager websites, the authors should explain clearly how training tasks are separated from evaluation tasks.

Questions for authors
  1. How does the Q-model accuracy change when the policy is updated? Have the authors evaluated Q predictions against actual success rates of newly generated trajectories?

  2. Can the authors provide a matched-compute comparison with trajectory-level RL, including environment collection cost and Q-model training cost?

  3. How are training and evaluation tasks separated on WebVoyager? Are there any overlapping instructions or trajectories between these two datasets?

Reviewer_YBU8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a step-level value model, agentic-Q, on trajectories collected with an SFT policy on live websites. Each step is supervised by the eventual outcome of the trajectory it belongs to. The resulting Q is then used inside SWPO, a step-level policy optimization procedure that combines candidate-action filtering with inverse-mean group weighting. On Ovis2.5-9B the method raises success rate from 68.68% to 78.79% on WebVoyager and from 52.67% to 64.00% on Online-Mind2Web, while grounding metrics stay essentially unchanged. An ablation shows that the full SWPO pipeline outperforms trajectory-level GRPO started from the same SFT initialization.

Critical strengths
  1. Substantial gains on live websites. The gains on live websites are substantial in absolute terms, and they are obtained offline with a frozen value model and no online interaction. On Online-Mind2Web the 9B model lands close to Claude 4 Sonnet CU, a gap large enough that it is hard to attribute to evaluation noise. That makes the case that the data-side treatment — trajectory stratification and environment-interference reruns — genuinely raised the usable fraction of SFT trajectories, which is where much of the practical difficulty in this setting actually sits. This is the main reason to take the empirical result seriously.
  2. Evaluation engineering that is documented and ablated. The paper also supports its evaluation engineering with ablations rather than mere assertion: interference detection with reruns, and the stratification of collected trajectories. These details are routinely left unexamined in live-web experiments even though they directly determine whether the reported numbers mean anything. Documenting them, and showing what each contributes, gives later work in this area something concrete to reuse.
Critical weaknesses
  1. agentic-Q shows no measurable gain of its own; all improvement comes from the two add-on mechanisms, and the reliability of the value model is never tested directly. In the ablation, step-level GRPO using agentic-Q alone reaches 73.35%, below the 74.51% of trajectory-level GRPO from the same initialization. Adding candidate filtering takes this to 76.46%, and adding inverse-mean group weighting to 78.79%. So step-level value supervision on its own does not beat the trajectory-level baseline — it is slightly worse, and a 1.16-point gap may well fall inside noise at this benchmark size — while the positive contribution is concentrated in two training-time devices whose benefit does not presuppose that Q is correct. The paper's central claim is that a step-level value signal improves policy optimization, and the ablation as reported points the opposite way. This is what keeps the submission below the acceptance threshold for me. The way Q is trained also cannot obviously support the kind of judgment that filtering requires. Within an SFT trajectory a state appears with exactly one action, and that action is labelled by the trajectory's final outcome, so what the model is fitted on is whether a step appeared in a winning trajectory — not which of several candidate actions at the same state is preferable. The within-state ranking ability that filtering depends on is never directly supervised. Q is additionally frozen throughout SWPO, so once the policy departs from the SFT initialization the candidates being scored fall outside its training distribution, and the paper does not report how far the policy actually drifts during training. Two benign explanations for the reported gains therefore remain open: filtering may act as a conservative regularizer simply by shrinking the candidate set, and Q's ranking may degrade on new states without this showing up in the aggregate numbers.
  2. The boundary with the closest prior work is not drawn. DigiRL and WebRL already train step-level value estimators from terminal outcomes with cross-entropy; Digi-Q already studies offline Q-learning with policy extraction; group-wise reward filtering is closely related to dynamic sampling as used in DAPO. None of these is cited or discussed, so the paper does not say what SWPO offers beyond them. Once weakness 1 removes the value model as the source of gain, the identifiable novelty narrows to the combination of candidate filtering with inverse-mean group weighting, and both components derive from existing ideas. Absent a point-by-point comparison, the case for acceptance rests on engineering composition plus live-web results. A table comparing against DigiRL, WebRL and Digi-Q along environment, online versus offline, reward source, base model and scale would settle this.
Questions for authors
  1. What is the success rate if candidate filtering keeps the same number of actions but selects them at random? This separates "Q's ranking is doing the work" from "shrinking the action space is doing the work."
  2. What does trajectory-level GRPO reach when given exactly the same filtering and weighting as SWPO, from the same SFT initialization? Whether that baseline reproduces something near 78.79% decides whether the advantage belongs to step-level value supervision or to the two add-on mechanisms.
  3. Please report how far the policy drifts from the SFT initialization during SWPO training (KL or an equivalent measure), and how Q's prediction error on the candidates it scores evolves over training. Calibration on held-out tasks, or within-candidate-set ranking accuracy, would be stronger still. Variance across runs would also help.
  4. How is inverse-mean group weighting defined for all-fail groups, where the group mean is zero? What fraction of each batch do such groups make up, and is the resulting gradient stable? Which group types contribute most of the final 2.33-point gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.