- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenges of training GUI agents in real-world environments, specifically the high cost of interaction and the issue of sparse rewards—where feedback is provided only at the end of a task (terminal rewards). The proposed method begins by collecting trajectories using an initial SFT (Supervised Fine-Tuning) policy and training an "agentic-Q" model based on terminal success or failure. This Q-model is then used to score actions sampled by the policy, facilitating single-step policy optimization. This approach eliminates the need for further real-world environment interactions during the reinforcement learning (RL) update phase. The authors report performance improvements over the baseline SFT policy on the WebVoyager and Online-Mind2Web benchmarks.
Critical strengths
-
The research motivation is clear. Reducing environment calls during policy training by leveraging a Q-model is a practical and valuable strategy for lowering the training costs of GUI agents.
-
The paper is well-written, the methodological diagrams are intuitive, and the overall training pipeline is easy to follow.
Critical weaknesses
-
The Q-model is trained using trajectories collected by the initial SFT policy and terminal rewards; consequently, its value estimates are heavily dependent on the subsequent performance of that SFT policy. For instance, an action that is actually conducive to task completion might receive a low score from the Q-model if the SFT policy fails to follow up on it effectively. Furthermore, §3.4 does not clarify whether the Q-model is updated during the policy training phase. If it remains static, there is a risk that estimation errors for actions sampled by the evolving policy will increase. The authors should provide empirical validation—for example, by comparing a static Q-model against a setup where trajectories are periodically re-collected and the Q-model is updated—while also reporting the associated additional interaction costs.
-
Digi-Q [1] has already implemented offline Q-guided policy training for GUIs and has compared Monte Carlo returns with TD learning. Therefore, using terminal returns to train a Q-function is not, in itself, a sufficient innovation. The paper should clarify what additional benefits its Q-estimation method and step-wise policy optimization provide over existing Q-guided approaches.
-
The manuscript does not cite or discuss Digi-Q. Given the similarity in the overall approach, I suggest adding a citation and explicitly detailing the methodological differences.
References
[1] Hao Bai et al. Digi-Q: Learning VLM Q-Value Functions for Training Device-Control Agents. ICLR, 2025.
Questions for authors
-
Does the Q-function remain fixed during SWPO? If it is updated, please explain the update mechanism and whether it requires additional environment interactions.
-
Have you verified whether, at different training stages, the Q-function’s ranking of candidate actions for a given state aligns with the actual outcomes of those actions when executed?
-
Compared to simply sampling actions using an SFT-trained policy and then using the Q-function to select the highest-scoring action, how much additional gain does the single-step policy optimization (SWPO) provide?