acceptodds
Under review as a conference paper at ICLR 2027

Beyond Coordinates: Semantic Reward for GUI Agent Post-Training

Abstract

Graphical User Interface (GUI) agents have made substantial progress in automating tasks with vision-language models. Reinforcement learning has further improved GUI grounding by optimizing policies with coordinate-based rewards, which provide direct supervision for spatial localization. However, GUI interaction data also contains rich semantic information about the intended operation, target element, and expected outcome, which cannot be fully captured by geometric objectives alone. To exploit this complementary information, we introduce semantic rewards for GUI agent post-training. During reinforcement learning, candidate actions are represented as natural-language descriptions of the intended GUI operations, and a VLM evaluates their semantic consistency with the target interaction. We further propose Self-Reward RFT, where the semantic evaluator is initialized from the policy and periodically updated with the latest policy weights while remaining frozen between updates. This design avoids maintaining a separately trained reward model while providing an adaptive semantic training signal. Extensive experiments across multiple GUI grounding benchmarks demonstrate the effectiveness of semantic supervision across different policy backbones. Using only 6k training samples, our framework with Self-Reward RFT improves MAI-UI-2B from 56.7% to 59.0% on the challenging ScreenSpot-Pro benchmark, achieving strong performance among lightweight GUI models while remaining competitive with several larger models. Further analysis shows that semantic and coordinate-based supervision capture complementary aspects of GUI action correctness, highlighting the value of incorporating semantic feedback into GUI agent post-training.

Primary area:
reinforcement learning
Keywords:
GUI Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NwqHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces semantic rewards for reinforcement learning of GUI agents. The policy describes its next action in natural language, and a vision-language evaluator rewards actions that are functionally equivalent to the target. Its Self-Reward RFT method periodically updates the evaluator from the policy while keeping it frozen between updates. With 6,000 training samples, MAI-UI-2B improves from 56.7% to 59.0% on ScreenSpot-Pro click grounding. The results suggest that semantic feedback complements coordinate-based rewards, although the additional gain from periodic synchronization is small and the evaluation focuses on single-step grounding rather than end-to-end task success.

Critical strengths
  1. The paper is clear and easy to follow.
  2. The semantic reward method is simple and effective, improving performance across multiple GUI grounding benchmarks.
  3. The ablation studies and analyses are thorough.
Critical weaknesses
  1. Self-Reward’s effect is unclear. Its gains over a fixed evaluator are only 0.1–0.3 points, with no paired multi-seed evidence that they exceed run-to-run variation. This weakens the paper’s central contribution.

  2. The headline comparison is not fully matched. The 6k semantic result is compared mainly with coordinate-reward training on 2k samples. Without a comparable larger-data baseline, it is unclear how much of the gain comes from the reward design.

Questions for authors

Introduction

  1. The motivation for Self-Reward is unclear. Could a value model, such as the critic used in PPO, achieve similar results?

Presentation

  1. Using AI to generate diagrams is understandable in the coding-agent era, but the authors should refine the output. Several labels in Figure 1 overflow their boxes. That said, the figure communicates the idea clearly at a glance.

Experiments

  1. The authors acknowledge that Self-Reward brings only small gains. Could these gains be due to random variation rather than the mechanism itself?
  2. Were the selected AndroidControl trajectories cleaned? To my knowledge, this dataset contains many cases where the instruction and ground truth are misaligned.
  3. Could the authors test Self-Reward with an 8B model and more training data to assess whether it scales?
Reviewer_HrkyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reward design for GUI agent reinforcement fine-tuning. It points out that existing methods primarily compute rewards based on the geometric alignment between predicted click coordinates and target locations, making it difficult to assess whether a predicted action is functionally consistent with the intended GUI operation. To address this limitation, the authors propose a semantic reward mechanism that represents model-generated actions as natural-language descriptions covering the target UI element, action type, and expected effect. A vision-language model (VLM) then evaluates the functional semantic consistency between the predicted and target actions, and its judgment is converted into a reward for group relative policy optimization (GRPO). Building on this approach, the authors further propose Self-Reward RFT, which initializes the semantic evaluator from the policy model and synchronizes its parameters with the policy every 50 optimization steps, keeping the evaluator frozen between updates. This design avoids the need to train and maintain a separate reward model. The method is evaluated on four GUI grounding benchmarks using two policy models, MAI-UI-2B and Qwen3-VL-2B, with 6K training samples. On ScreenSpot-Pro, for example, the method improves the accuracy of MAI-UI-2B from 56.7% to 59.0% and that of Qwen3-VL-2B from 43.4% to 45.7%. The authors also conduct ablation studies to examine the contributions of semantic and coordinate-based rewards and analyze the complementarity of these two signals in evaluating GUI action correctness.

Critical strengths
  1. Exploring semantic supervision for GUI action optimization. The paper investigates the use of semantic consistency rewards to complement conventional coordinate-based rewards in GUI post-training. The ablations show that semantic rewards can provide additional gains when coordinate-based rewards yield limited improvements, particularly for the GUI-specialized MAI-UI-2B (Table 3). The method also enables the use of action-prediction samples without coordinate annotations, offering a potential way to leverage training data that cannot be directly used by coordinate-based reward formulations (Table 5). The comparison with SFT on the same semantic-action samples further provides evidence that the observed gains are not simply attributable to additional language supervision (Table 8).
  2. Useful empirical evidence on evaluator efficiency. The reward-model comparison shows that lightweight GUI-specialized evaluators can outperform a larger general-purpose VLM while requiring substantially less compute. Specifically, MAI-UI-2B and UI-Venus-1.5-2B achieve scores of 58.4 and 58.7 with 48 and 43 GPU hours, respectively, compared with 57.6 for Qwen3-VL-8B using 132 GPU hours (Table 4). This result provides practical evidence that domain specialization can be an effective alternative to scaling up general-purpose evaluators for semantic reward evaluation.
Critical weaknesses
  1. Marginal empirical gains from periodic weight synchronization in Self-Reward RFT. Although periodic synchronization of the semantic evaluator with the evolving policy every 50 steps is presented as the second main contribution, its empirical gains are marginal. Table 3 shows that enabling synchronization (Self-Reward ✓ vs. ✗) improves MAI-UI-2B from 58.4 to 58.5 (+0.1 points) and Qwen3-VL-2B from 45.4 to 45.7 (+0.3 points). Moreover, Figures 5–6 show that the evaluator's accuracy (around 0.80) and AUROC (0.865 to 0.869) remain nearly unchanged from initialization through step 150. Thus, the empirical evidence supports reusing the initial policy as a frozen zero-cost evaluator, but does not convincingly demonstrate that periodic weight synchronization provides a meaningful improvement; moreover, no ablation on the synchronization interval is provided.
  2. Mismatch between semantic action training and coordinate-only evaluation, with regressions on text-dense interfaces. During semantic RFT, the policy is instructed to generate semantic actions rather than coordinates, yet all four evaluation benchmarks focus exclusively on single-step click-coordinate grounding. The paper does not evaluate multi-step GUI tasks or action-prediction benchmarks to establish whether semantic rewards improve action selection or task completion, nor does it analyze how language-based RL supervision transfers to coordinate grounding. Moreover, the semantic verifier is vulnerable to lexical overlap between distinct UI controls, achieving only ~0.55 accuracy on Text-lex overlap (Figure 3). This issue may contribute to the observed regressions on text-heavy subsets, including a 7.3-point drop for Qwen3-VL-8B on Office Text (85.3 to 78.0). These results leave both the broader applicability of semantic rewards and their reliability on text-dense interfaces insufficiently established.
  3. Insufficient details on instruction synthesis for GroundCUA. Section 5 claims that semantic supervision can be obtained from existing datasets without additional annotation. However, adapting GroundCUA, which accounts for half of the 6k training set, requires using GPT-5.5 to infer high-level task instructions from low-level annotations (Appendix D, p. 20). Although the paper refers to Appendix D for construction details, it does not provide the synthesis prompt, quality-control procedure, or associated costs. These omissions make it difficult to assess the reproducibility and practical overhead of the data construction process.
  4. Minor presentation issue (Figure 1). Some text in Figure 1 extends beyond its designated boxes, resulting in visual inconsistencies.
Questions for authors
  1. Benefit and sensitivity of periodic weight synchronization: How sensitive is performance to the synchronization interval (e.g., frozen initial policy vs. synchronizing every 10, 50, or 100 steps), and does periodic synchronization provide a statistically meaningful gain over simply keeping the initial policy frozen as the evaluator throughout training?
  2. Action-prediction evaluation and regressions on text-dense interfaces: Since semantic RFT trains the policy to output natural-language low-level actions without coordinates, how does this supervision improve downstream action-prediction accuracy (e.g., action-type and step-level accuracy on the AndroidControl test split)? In addition, what causes the regressions on text-heavy grounding subsets in Table 1—most notably the 7.3-point drop on Office Text for Qwen3-VL-8B (85.3 to 78.0) and the 2.6-point drop on Dev Text for MAI-UI-2B (78.6 to 76.0)?
  3. Reproducibility of GPT-5.5 instruction synthesis for GroundCUA: Could you provide the exact prompt, contextual inputs, filtering criteria, and API cost used to synthesize high-level task instructions for the 3k GroundCUA samples with GPT-5.5?
Reviewer_FQDJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains the GUI grounding model to describe the next action; a VLM judges the descriptions to determine whether they are functionally equivalent to the target action. The self-reward method uses the model itself as the judge and synchronises the weights periodically during training. The model based on MAI-UI-2B and Qwen3-VL-2B demonstrates improvement on ScreenSpot-Pro.

Critical strengths

The method that rewards the model for generating a correct action description is well-motivated and makes sense. The experiments on two base models demonstrate the effectiveness of the method. The paper compares coordinate and semantic rewards, evaluator choices, and semantic SFT with RL. The human-labeled equivalence set also tests the reward signal directly.

Critical weaknesses
  1. Prior studies have proposed supervising natural-language descriptions of intended actions in related and broader domains [1–4], which limits the novelty of the proposed method.

  2. The semantic reward’s notion of functional equivalence is insufficiently validated against actual GUI behavior. Figure 10 explicitly treats clicking a search box and clicking a search icon as equivalent and instructs the verifier to be lenient. However, these actions can produce different transitions, for example, focusing an input field versus submitting a query.

[1] Embodied Navigation with Auxiliary Task of Action Description Prediction [2] MobiAgent: A Systematic Framework for Customizable Mobile Agents [3] RT-H: Action Hierarchies Using Language [4] Joint Action Language Modelling for Transparent Policy Execution

Questions for authors

Do the gains over coordinate-based RL baselines also hold for larger policy backbones (e.g., 4B–8B)? Could you provide controlled ablations at these scales? Does the method generalize to additional GUI benchmarks, such as UI-Vision? How does the model perform on more general vision-language benchmarks after training? Could you provide results to assess this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.