acceptodds
Under review as a conference paper at ICLR 2027

Group-Relative Self-Verified Policy Optimization for Label-Free GUI Grounding

13 views

Abstract

Reinforcement learning for GUI grounding commonly relies on binary rewards that indicate whether a predicted click falls inside an annotated target region. Such rewards become uninformative when all rollouts in a group share the same outcome, and they also collapse near misses and distant errors into the same supervision. Our analysis further reveals that many near misses are produced with high confidence, suggesting that models often identify the approximate target region but lack sufficient clicking precision. These observations suggest that the missing supervision is not necessarily additional labels, but a graded signal that can exploit the model's own approximate localization. Motivated by this, we propose Group-Relative Self-Verified Policy Optimization (GSVPO), which converts local re-observation and cross-trajectory self-verification into a continuous, label-free grounding reward. GSVPO samples multiple global click proposals, re-examines each proposal under an entropy-adaptive magnified view, and maps the refined predictions back to the original coordinate space. Candidates are then evaluated according to their agreement with the ensemble of refined predictions across trajectories, preserving graded spatial distinctions without requiring ground-truth coordinates. The resulting signal provides a unified interface for training-free candidate reweighting, label-free test-time reinforcement learning, and reward augmentation for standard GRPO. Across six GUI grounding benchmarks and six vision-language models, training-free GSVPO improves 32 of 36 model–benchmark pairs, with an average gain of 6.03 percentage points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agent, GUI Grounding, Reinforcement Learning, GRPO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qaKqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the limitations of binary rewards in GUI grounding. It proposes GSVPO, which uses entropy-guided local re-observation and agreement across trajectories to produce a continuous, label-free reward. The authors evaluate it for training-free inference, test-time reinforcement learning, and reward augmentation.

Critical strengths
  1. A scalable label-free training signal. Building on the idea of test-time reinforcement learning, the paper turns the model’s approximate localization ability into a graded reward without ground-truth coordinates. This could reduce reliance on costly GUI target annotations as training data scales.

  2. Broad empirical gains. The method improves grounding across several GUI-specialized models and benchmarks, with particularly clear gains on challenging interfaces. This suggests the approach is useful beyond a single model architecture.

Critical weaknesses

The contribution of cross-trajectory verification is not isolated. In Table 3, selecting only zoomed-in clicks achieves 33.1% on ScreenSpot-Pro, compared with 33.2% for full GSVPO. Without a matched comparison that removes the consensus reward while retaining local re-observation, it is unclear whether the paper’s central mechanism drives the inference gains. The missing latency and cost measurements further limit the practical assessment of this test-time method.

The evaluation does not adequately establish the advantage over the closest alternatives. The paper does not directly compare its label-free adaptation with Learning from Reliable Negatives, or its response to binary-reward saturation with online filtering such as DAPO. It also does not clearly distinguish its motivation from prior dense, ground-truth-based rewards. These gaps make it difficult to assess the novelty and necessity of GSVPO relative to existing approaches.

Questions for authors

Introduction

  1. The paper states that most prior methods use binary click rewards, but SE-GUI (NeurIPS 2025) and GUI-G² (AAAI 2026) propose distance-based and Gaussian dense rewards. How does GSVPO’s motivation differ from these approaches?
  2. The paper argues that static data filtering becomes ineffective as the policy changes. Could online filtering, as used in DAPO, address this issue?
  3. Is the claim about how humans inspect and verify difficult clicks supported by empirical research?
  4. What evidence supports using coordinate-token entropy to determine crop size across targets of different sizes?

Related work

  1. The label-free grounding discussion omits the closely related ECCV 2026 paper [1]. Please explain how GSVPO differs from it.

Presentation

  1. Figure 2 does not appear to be cited in the text. Its labels “Q” and “S” are also unclear, since the method defines the screenshot–instruction pair as (I,q)(I,q).

Experiments

  1. As a test-time scaling method, GSVPO needs a systematic best-of-NN direct-sampling baseline under a matched computation budget.
  2. Please report inference latency and cost. GSVPO generates multiple full-screen proposals and performs a local prediction for each, but the paper does not clearly quantify the added model calls or accuracy–cost trade-off.

[1] Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding.

4 comments

Thank you very much for your patience with our work and for your valuable suggestions. We will address your concerns one by one below.

W1. Cross-trajectory verification is not isolated and Q4. Experiment 2 (latency and cost). Thank you for this suggestion. We agree that the previous zoom-only row retained the cross-trajectory reward. However, in our "Single-traj zoom-in click" and "Single-traj full-screen click" experimental designs, cross-trajectory consensus is directly isolated. These two experiments are designed to consider only each trajectory's individual local re-observation under the same settings as GSVPO, without considering the consensus among multiple samples brought by cross-trajectory information. Their final accuracies are 26.1% and 19.3%, respectively. We add a matched ablation using exactly the same global proposals, crops, local re-observations, and zoom-candidate output space, while setting the self-verification advantage to zero. Selecting zoom candidates only by their native local sequence scores obtains 27.6%, whereas adding the cross-trajectory advantage obtains 33.2% (+5.6 points). A single greedy click followed by one local re-click obtains 26.7%, which is still lower than the GSVPO result. Therefore, whether considering the performance gains from local re-observation under high-temperature sampling or from local re-observation under greedy clicking, the comparison with GSVPO demonstrates the effective improvement brought by the cross-trajectory reward. These results separate the gain from magnification itself from the additional gain provided by cross-trajectory verification. We measured latency and computational cost on the same fixed ScreenSpot-Pro samples. Each sample was processed on one RTX 4090; multiple GPUs only parallelized the benchmark. Latency includes preprocessing, generation, decoding/scoring, and selection, while excluding model loading, warm-up, and disk I/O.

SettingCandidate clicksGenerationsMean latencyP90 latencyInput tokensOutput tokens
Direct prediction112.411 s4.351 s8.19K13.0
Four trajectories89*19.543 s32.085 s73.72K115.6
Full GSVPO181838.564 s62.827 s147.44K231.2

*The four-trajectory setting uses eight candidate generations plus one entropy-estimation call.

P90 latency denotes the 90th-percentile per-example latency: 90% of the evaluated examples finish within this time.

For full GSVPO, cross-trajectory reward computation and candidate selection require only 0.135 ms per sample; nearly all additional latency comes from the VLM generations. Its average peak allocated GPU memory is 8.45 GiB. Therefore, Table 3 variants that reuse the same 18 candidates have identical neural inference cost and differ only in negligible selection overhead. We will add these measurements and explicitly discuss this accuracy-efficiency trade-off.

W2. Advantage, difference from other methods. and Q1.Introduction 2 (Difference from DAPO). First, our evaluation demonstrates that magnification-based self-verification yields significant gains on the more challenging ScreenSpot-Pro, and our method outperforms comparable methods such as GUI-RC, BAMI, and UI-Zoomer. Second, in test-time RL, our method achieves better generalization on unseen data, outperforming R-OPSD on ScreenSpot-Pro, OSWorld-G, and OSWorld-G-R, and surpassing GUI-RCPO in the vast majority of comparison cases, which sufficiently demonstrates our performance advantage. Second, our method aims to obtain richer training information without ground truth through self-verification. Through careful search and reading, we consider GUI-RC, GUI-RCPO, and R-OPSD to be methods of the same type as ours, so we discussed them and conducted experimental comparisons, which is sufficient. The paper "Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding" was released on September 14, 2026, only four days before the ICLR 2027 abstract deadline; using this paper to argue that our comparison is insufficient would be unreasonable. Our search and comparison are sufficiently thorough, and regarding DAPO, the overlap with DAPO lies in the motivation, not the mechanism. DAPO retains binary rewards, discards all-correct or all-wrong groups, and resamples until enough informative groups are obtained; discarded rollouts provide no gradient despite their generation cost. GSVPO instead retains these groups and derives a continuous, label-free advantage from cross-trajectory spatial agreement, extracting relative preferences even when all binary rewards are identical. DAPO is therefore a batch-sampling strategy requiring an external correctness signal, whereas GSVPO is a reward-construction mechanism and is orthogonal to DAPO. To prevent similar confusion among future readers, we will add a discussion of the differences in the appendix. Finally, you believe that we have not distinguished our method from other types of ground-truth-based rewards, but our primary starting point is the label-free reward setting, where the reward comes from the model's own magnification-based verification. This starting point is well-grounded (supported by the analytical experiments at the beginning of the paper) and is novel in the area; the inspiration for cross-trajectory self-verification comes from human intuition when identifying complex click targets, as well as from the phenomena discovered and conjectures formed in our earlier analytical experiments.

Q1. Introduction 1 (Difference from SE-GUI, GUI-G2). Thank you for this important distinction. The key difference is the source of supervision. SE-GUI computes distance to the annotated target, while GUI-G² centers its Gaussian point and coverage rewards on the ground-truth element. Their dense rewards therefore still require target coordinates or bounding boxes. Moreover, we have never stated in the paper that these frontier methods rely solely on binary rewards. What we said is that although some methods have proposed different rewards, most of them still need binary rewards to provide training signals to varying degrees. In these methods, hit and miss is an important supervisory signal, and many methods rely on ground-truth labels. This is where GSVPO differs from them. GSVPO does not use the ground-truth target. It estimates spatial preference from agreement among independently cropped and mapped-back local predictions. This enables the same reward to support label-free test-time training and frozen-policy candidate selection, where ground-truth-based rewards cannot be computed. Thus, our contribution is not the Gaussian kernel itself, but label-free cross-trajectory local self-verification as the source of the dense reward.

Q1. Introduction 3 (whether empirical research supports human behavior). We mention the ways humans might approach complex problems only as a starting point for the genesis of our idea. We drew inspiration from this possible human behavior and then developed our method. The strong basis for our method lies in the phenomena we observed in Fig.1(a) and (b), from which the idea emerged, and we subsequently validated it with three RQs and experiments on six benchmarks. Our experiments are sufficient. We never claim in the paper that the effectiveness of our method depends on this human behavior; it is merely a statement to help readers better understand our idea and where it came from.

Q1. Introduction 4 (entropy to crop size). Thank you for this question. We use coordinate-token entropy as a model-intrinsic measure of localization uncertainty, rather than as a direct estimator of physical target size. This interpretation is supported by recent studies. Li et al. [1] show that high-entropy tokens in visual grounding are concentrated on predicted coordinates. Luo et al. [2] further demonstrate that uncertainty computed from generated-token distributions correlates with the preferred visual resolution, while Gröpl et al. [3] use next-token entropy gradients to retrieve visual evidence and guide iterative zooming. Our specific entropy-to-crop mapping is validated empirically. On ScreenSpot-Pro, a 0.3 x 0.3 crop improves over 0.5 x 0.5 by 1.9 points, whereas 0.7 x 0.7 reduces accuracy by 4.8 points. At fixed area, extending the crop along the higher-entropy coordinate axis improves accuracy by 3.2 points, while compressing that axis reduces it by 1.4 points. Every design mentioned in our method is supported by experiments. The same holds for this design, and controlled-variable experiments demonstrate its effectiveness.

[1] The Role of Entropy in Visual Grounding: Analysis and Optimization, Findings of EMNLP 2026

[2] Task-Aware Resolution Optimization for Visual Large Language Models, EMNLP 2025

[3] Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models, ECCV 2026

Q2. Related Work (Difference from an ECCV work). Thank you for pointing out this concurrent work. [1] selects the full-image prediction with the highest coordinate-token confidence as a single pseudo-label, assigns binary rewards by distance to this anchor, and updates only reliable negative samples. GSVPO uses coordinate entropy only to determine the local observation scale; it re-observes every proposal, maps all refined clicks back to the original image, and derives a continuous reward from cross-trajectory self-verified agreement. Thus, GSVPO does not trust any single prediction as a pseudo-label; it attempts to break stubborn errors and error propagation across trajectories and to obtain fine-grained information through magnification-based self-verification. It additionally supports training-free reweighting, label-free adaptation, and reward augmentation. Therefore, I think they are different in the starting point of the method design, the specific sub-problem they try to solve, the design of the reward source, and the definition of the problem, and there is even a certain gap between them. In contrast, GUI-RC, GUI-RCPO, and R-OPSD, which are repeatedly mentioned in the paper, are more worth discussing and are works that walk on a similar research path. Importantly, [1] first appeared on arXiv on September 14, 2026, only four days before the September 18 ICLR 2027 abstract deadline. It was therefore concurrent work that was not reasonably available for a full comparison during submission, and its absence should not be interpreted as overlooking established prior work.

We will cite it and clarify these distinctions in the revision.

[1] Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding.

Q3. Presentation. Thank you for noting this presentation issue. In Fig. 2, “Q” denotes the user query/instruction and “S” denotes the agent’s visual state, i.e., the current screenshot. We originally considered this query–state notation to be common in agent literature and therefore treated it as a convention in the field, but to prevent similar confusion, we will revise it accordingly. We will replace “S” and “Q” with “I” and “q”, respectively, and explicitly cite Fig. 2 in the method overview.

Q4. Experiments 1 (Best-of-(N)). We thank the reviewer for suggesting this control. We compare GSVPO with direct best-of-N sampling using the same model, sampling configuration, examples, and total number of generations. Direct BoN samples exclusively from the full screenshot and selects by native sequence score, whereas GSVPO divides the same budget between full-image proposals and paired local re-observations. We additionally report Direct oracle as a non-deployable upper bound on full-image candidate coverage.

MethodN=1N=2N=4N=8N=18
Direct native-score BoN13.013.413.813.815.0
Direct oracle13.014.217.422.027.0
GSVPO-23.626.829.033.2

These compute-matched results show that the improvement cannot be explained by additional sampling alone. Local re-observation produces better candidates, while cross-trajectory verification selects them effectively.

Reviewer_CpexOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GSVPO addresses the limited feedback available for GUI grounding when coordinate annotations are unavailable or binary rewards cannot distinguish candidate clicks. It uses local re-observation and agreement among refined predictions to construct a graded self-verification signal, supporting candidate selection, test-time learning, and augmentation of supervised reinforcement learning. Evaluation across multiple models and grounding benchmarks reports broad improvements, while sampling controls and transfer experiments examine whether the benefits extend beyond generating more candidates or adapting to a single benchmark.

Critical strengths
  • Improvements are broad: Table 1 reports gains in 32/36 settings. Table 3 also addresses sampling budget: four global-plus-local trajectories achieve 29.0% on ScreenSpot-Pro versus 14.7% for eight full-image clicks, supporting a benefit beyond merely generating more candidates.
  • Table 2 includes direct-click evaluation after adaptation, with OSWorld-G gains of 14.2 and 13.8 points for the 3B and 7B models. This supports learning that transfers beyond the adaptation benchmark and does not require cropping during evaluation.
Critical weaknesses
  • Equations (8)–(11) score every candidate under the full-image policy, although local candidates originate from cropped inputs and transformed coordinates (Eq. (2)). Section 4.3 instead describes updating crop-conditioned clicks. Appendix A.2 and Algorithm 1 do not resolve which likelihoods are actually optimized. This matters for reproducing the reinforcement-learning contribution and interpreting its policy-gradient formulation.
  • The claimed complementarity with binary supervision rests on a 0.8-point gain on one benchmark (Fig. 3(a)), with all experiments using seed 42 (Appendix A.2). This narrow result provides limited support for the general complementarity claim.
Questions for authors
  1. Clarification: Are local candidates rescored under the full screenshot or optimized under their original crop? Please specify the corresponding coordinates, likelihoods, and probability ratios, and how the objective handles their generation procedure.
  2. How stable is the 31.4% versus 30.6% reward-combination improvement across training seeds?
3 comments

Thank you very much for your patience with our work and for your valuable suggestions. We will address your concerns one by one below.

W1 and Q1. Explain how the equations match. Thank you for identifying this notation ambiguity. Local candidates are generated, sequence-scored, and optimized under their original crops; they are not rescored under the full screenshot.

For each base response, we have:

yibase∼πθ(⋅∣I,q),ai=Decode⁡(yibase).y_i^{\mathrm{base}}\sim\pi_\theta(\cdot\mid I,q),\qquad a_i=\operatorname{Decode}(y_i^{\mathrm{base}}).

The base proposal determines a crop, from which a separate local response is generated:

Iiloc=C(I;ai,w,h),yiloc∼πθ(⋅∣Iiloc,q).I_i^{\mathrm{loc}}=C(I;a_i,w,h),\qquad y_i^{\mathrm{loc}}\sim\pi_\theta(\cdot\mid I_i^{\mathrm{loc}},q).

The decoded local coordinate is initially expressed in the crop coordinate system and is mapped to the original screenshot only for reward computation and final selection:

v~i=Decode⁡(yiloc),vi=Mi−1(v~i).\tilde v_i=\operatorname{Decode}(y_i^{\mathrm{loc}}),\qquad v_i=M_i^{-1}(\tilde v_i).

Accordingly, Eq. (8) uses each response's native sequence likelihood:

z′(ai)=log⁡πθ(yibase∣I,q)+βAsv(ai).z'(a_i)=\log\pi_\theta(y_i^{\mathrm{base}}\mid I,q)+\beta A_{\mathrm{sv}}(a_i).

z′(vi)=log⁡πθ(yiloc∣Iiloc,q)+βAsv(vi).z'(v_i)=\log\pi_\theta(y_i^{\mathrm{loc}}\mid I_i^{\mathrm{loc}},q)+\beta A_{\mathrm{sv}}(v_i).

Likewise, Eq. (9) optimizes each response under its native observation:

LSV=−12K∑[ASV(a(i))log⁡π(ybase,(i)∣I,q;θ)+ASV(v(i))log⁡π(yloc,(i)∣Iloc,(i),q;θ)].\mathcal{L}^{\mathrm{SV}}=-\frac{1}{2K}\sum\left[A^{\mathrm{SV}}(a^{(i)})\log\pi\left(y^{\mathrm{base},(i)}\mid I,q;\theta\right)+A^{\mathrm{SV}}(v^{(i)})\log\pi\left(y^{\mathrm{loc},(i)}\mid I^{\mathrm{loc},(i)},q;\theta\right)\right].

In Eq. (11), the current and old policies evaluate the same stored response under the same native observation. The token-level probability ratios are:

ri,tbase=πθ(yi,tbase∣I,q,yi,1:t−1base)πθold(yi,tbase∣I,q,yi,1:t−1base).r_{i,t}^{\mathrm{base}}=\frac{\pi_\theta(y_{i,t}^{\mathrm{base}}\mid I,q,y_{i,1:t-1}^{\mathrm{base}})}{\pi_{\theta_{\mathrm{old}}}(y_{i,t}^{\mathrm{base}}\mid I,q,y_{i,1:t-1}^{\mathrm{base}})}.

ri,tloc=πθ(yi,tloc∣Iiloc,q,yi,1:t−1loc)πθold(yi,tloc∣Iiloc,q,yi,1:t−1loc).r_{i,t}^{\mathrm{loc}}=\frac{\pi_\theta(y_{i,t}^{\mathrm{loc}}\mid I_i^{\mathrm{loc}},q,y_{i,1:t-1}^{\mathrm{loc}})}{\pi_{\theta_{\mathrm{old}}}(y_{i,t}^{\mathrm{loc}}\mid I_i^{\mathrm{loc}},q,y_{i,1:t-1}^{\mathrm{loc}})}.

During optimization, each response and its native observation are treated as a fixed pair. The sampled base proposal, crop construction, coordinate mapping, and advantages are detached; no gradient passes through the discrete crop operation. Therefore, the base and local responses are optimized as separately conditioned policy sequences rather than as a concatenated two-action trajectory.

Eq. (10), which only combines advantages, is unchanged. We will revise Algorithm 1, and Appendix A.2 to explicitly specify the coordinates, native conditioning observations, likelihoods, and probability ratios.

W2.Limited support. We acknowledge that the final improvement of only 0.8 is not particularly impressive. However, as can be seen from the curve in Fig. 3(a), as the performance improvement of GRPO under binary rewards gradually levels off, the gap begins to widen, and this curve behavior is more objective than simply looking at the final performance. Furthermore, when designing this experiment, to make it more challenging, we chose pre-filtered mixed samples, as you can see in the Appendix. For binary rewards, such samples can provide more effective training gradients in the early stage. If we had chosen all-wrong sample groups, the model's performance when using only binary rewards in the early stage would be difficult to improve because most groups would have zero gradient, and the experiment would then lose its meaning, merely overlapping with the conclusion supported by our previous experiments, namely that GSVPO can provide effective gradients in the label-free setting. Therefore, a drawback of this choice is that the final accuracy improvement is not sufficiently pronounced. To demonstrate the stability of the performance improvement across multiple seeds, we added the gains for seeds 40 and 44 as shown in the table, yielding an average gain of 1.1%, which shows that our effect has a certain degree of consistency. To clarify this choice, we will add some discussion in the appendix and change our wording from "complementary" to "exhibit partial complementarity" to better align with the experimental results.

seed: 40seed: 42seed: 44
SS-Pro Gain (%)+1.0+0.8+1.5

Q2.Across-seeds experiment. As shown in the table, although the gains are not pronounced, a consistent gain does exist; the rationale for this experimental design is explained in W2.

I see, thank you for the clarification. I think my concerns are well addressed, I keep my positive score.

Reviewer_1MYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GSVPO stands for Group-Relative Self-Verified Policy Optimization. It first predicts clicks from the full screenshot. It then makes new predictions in entropy-adaptive magnified views. Cross-trajectory self-verification scores each candidate by how well it agrees with the local predictions. The score is used for training-free inference, test-time reinforcement learning, and reward design for policy optimization. The training-free tests improve 32 of 36 model–benchmark pairs, by an average of 6.03 percentage points.

Critical strengths
  1. The reported gains cover several models and benchmarks. Table 1 tests six models on six benchmarks. The gains are especially large on ScreenSpot-Pro. Table 3 also compares four global/local trajectories, with eight clicks in total, against eight full-screen clicks: accuracy is 29.0% versus 14.7%. The gain is therefore not explained only by producing more clicks.
Critical weaknesses
  1. The tests do not clearly separate better candidates from a better score. In Table 3, using only zoomed candidates gives 33.1%, close to 33.2% with the full pool. Both still use the proposed score. A test of different scores on the same candidate pool is needed. The comparison with GUI-RC also needs clarification. Its published Qwen2.5-VL-3B result on ScreenSpot-V2 is 80.11→82.63. This paper starts from 62.7 but lists the same rounded gain of +2.5. Gains from different starting points do not establish which method is better.
  2. The claims about label-free learning and unseen data need limits. Appendix A.2 uses all 1,272 ScreenSpot-V2 inputs for test-time reinforcement learning. Evaluation on those inputs is not a test on unseen inputs. ScreenSpot-V2 is also a revision of ScreenSpot, so ScreenSpot is not an independent unseen set after this training. In RQ3, labels are used to select the 1,500 mixed examples, even though the SV-only reward does not use labels. The paper should also explain what data were used to choose the benchmark-specific crop ranges.
Questions for authors
  1. Clarification: what data were used to choose crop ranges and other settings? Please separate ScreenSpot/V2 results from tests on independent unseen inputs.
  2. Clarification: is a local candidate rescored using the full screenshot, scored using its crop, or scored as part of a whole trajectory? Please explain how this matches Equations (8–11) and the whole-trajectory GRPO description in the appendix.
4 comments

Thank you very much for your patience with our work and for your valuable suggestions. We will address your concerns one by one below.

W1.Test can be improved. Thank you for this suggestion. We added a matched ablation using exactly the same candidates (full-screenshot and locally re-observed clicks), changing only the selection score. Selecting by the native sequence score ziz_i achieves 27.6%, while GSVPO selection using zi+βAsv(ui)z_i+\beta A_{\mathrm{sv}}(u_i) achieves 33.2% on the same ScreenSpot-Pro samples as shown in the table. Thus, with local re-observation, candidate pool, and inference cost fixed, cross-trajectory verification provides a +5.6 percentage-point gain. We understand that directly comparing net gains may not be entirely appropriate when the starting points differ. For GUI-RC, we carefully followed the released implementation and evaluated Qwen2.5-VL-3B under our standardized setting, including the same prompt template, preprocessing, decoding configuration, and evaluation protocol used throughout our experiments. Under this setting, we consistently obtained a baseline accuracy of 62.7% on this benchmark. We therefore report improvements relative to this reproduced baseline, which provides a controlled and internally consistent comparison, rather than directly comparing final scores obtained from different starting points and evaluation settings. Considering that comparisons among methods of a similar type to ours need to highlight the gains, we therefore chose to report net gains. As for the discussion of this result, we believe that taking different starting points into account as a condition in the analysis is a more appropriate approach. Therefore, in the subsequent stage, we will add this content to the experimental section or the corresponding appendix to explain this situation, and to clarify that our method does not use this metric to prove that it is absolutely better than theirs, but only to provide a reference under different starting points. We will soon revise the PDF.

ziz_izi+βAsv(ui)z_i+\beta A_{\mathrm{sv}}(u_i)
SS-Pro Acc. (%)27.633.2

W2. Claims about label-free and unseen data. For a fairer comparison, we follow the training set choices of prior work, GUI-RCPO and R-OPSD, training on ScreenSpot-V2 and using ScreenSpot-V2 and ScreenSpot as test sets. Therefore, like these works, ScreenSpot-V2 is not intended to verify the model's label-free training performance on unseen data but rather serves as a reference. The remaining four benchmarks besides ScreenSpot and ScreenSpot-V2 can all serve as experimental results on unseen data, and the experimental volume on these four benchmarks is sufficient to support this conclusion. For better understanding, we will emphasize this point in the appendix and soon revise the PDF. In addition, regarding the label-free data selection in RQ3, first, it should be noted that our RQ3 is intended to investigate a research question. One important comparison group in our experiments is naive GRPO under binary rewards. Therefore, filtering out mixed samples is to demonstrate that even after mixed samples are selected, binary rewards still face the risk of degradation, and the SV reward can still help enhance binary rewards. Second, as we stated at the beginning, the main contribution of the label-free reward setting lies in continuously providing reward gradients and offering richer training signals. Even after using the earlier 8-click test for filtering, this data still faces the degradation problem in which within-group differences gradually become zero. Therefore, this data is entirely suitable as a research object for RQ3. As a research question, the role of the dataset in investigating this research question is to serve as experimental samples that support our claims, and this dataset fully meets this requirement. Moreover, we have never said that the purpose of RQ3 is to train a model to achieve higher accuracy (we do not demonstrate the effect of label-free training through better final accuracy); it is to explain that the SV reward can provide more effective training signals for ordinary GRPO binary rewards, which is consistent. Finally, our algorithm itself is label-free, and the keyword "label-free" fully matches our algorithm design; it is only under this research question that we selected this data to better illustrate our point. To address potential confusion among future readers, we will clarify this point in the appendix.

W2 and Q1. Crop size setting. These two questions concern the same missing implementation detail. No labeled development or evaluation data were used to select the benchmark-specific crop ranges. We fixed two coarse presets before evaluation based only on the benchmark descriptions in the original papers. Benchmarks characterized by high-resolution interfaces with tiny and densely packed targets, such as ScreenSpot-Pro, use a stronger-magnification range of [0.04,0.25][0.04,0.25]; benchmarks described as containing larger and easier-to-localize targets use a more context-preserving range of [0.4,0.7][0.4,0.7]. These ranges were fixed across all evaluated models within each benchmark. The crop-range experiments in Fig. 3(c–d) are sensitivity analyses conducted to examine these fixed choices. For unseen data, as we discuss in W2, we will add this selection protocol and distinguish the manually specified crop presets from the sample-wise entropy adaptation in Appendix A.2. We will discuss this issue in an appropriate place and rigorously constrain the source of the generalization evidence on unseen data to the results on the other four benchmarks.

Q2. Explain how the equations match. Thank you for pointing out this ambiguity. A base proposal and its local refinement form a verification pair for reward construction, but they are not optimized as one concatenated two-action trajectory:

\qquad \tilde v_i \sim \pi_\theta(\cdot \mid C(I;a_i,w,h),q), \qquad v_i=M_i^{-1}(\tilde v_i).$$ Thus, the local response is generated and sequence-scored on its crop, and is never rescored on the full screenshot. After mapping, all candidates form the common pool $\mathcal U=\mathcal A\cup\mathcal V$ and receive individual cross-trajectory advantages $A_{\mathrm{sv}}(u)$. Let $x_u$ be the native observation of candidate $u$ (full screenshot for a base candidate and its crop for a local candidate), and let $y_u$ be its generated response. Its native sequence score is $$z(u)=\sum_t \log \pi_\theta(y_{u,t}\mid x_u,q,y_{u,<t}).$$ Accordingly, Eq. (8) uses $$z'(u)=z(u)+\beta A_{\mathrm{sv}}(u),$$ while Eq. (9) optimizes $$\mathcal{L}^{\mathrm{SV}}(\theta)=-\frac{1}{|\mathcal{U}|}\underset{u\in\mathcal{U}}{\sum}A^{\mathrm{SV}}(u)\log\pi\left(y^{(u)}\mid x^{(u)},q;\theta\right).$$ Here, the probability covers the complete generated response: $$\log\pi_\theta(y_u\mid x_u,q)=\sum_t\log\pi_\theta(y_{u,t}\mid x_u,q,y_{u,1:t-1}).$$ Eq. (11) uses the same native response and observation for both the current and old policies, while Eq. (10) only combines the advantages and is unchanged. Here, “whole-trajectory” means applying each candidate’s advantage to its complete response sequence; a candidate's scalar advantage is applied to its complete generated response sequence, rather than only to its coordinate tokens. It does not mean jointly optimizing $(a_i,v_i)$. Therefore, $A_{\mathrm{sv}}(v_i)$ updates only the crop-conditioned local response, with no gradient through the crop to $a_i$. We will revise the appendix to make this explicit.

Comments on the paper

3 comments

Thank you very much for the reviewers' valuable comments. If you have any further questions about our responses and clarifications, please feel free to raise them and discuss them further so that you can better understand our work. If you are satisfied with our responses, please also consider increasing our score. Thank you again for your time and patience.

Thank you very much for the reviewers' valuable feedback. Should you have any remaining questions about our responses or clarifications, please do not hesitate to raise them so that we can discuss them further and help you gain a clearer understanding of our work. If our responses have addressed your concerns, we would greatly appreciate it if you would consider raising our score. Thank you once again for your time and patience.

Thank you very much for the reviewers' valuable feedback. We would be happy to address any remaining questions about our responses or clarifications; please feel free to raise them so that we can discuss them further and ensure our work is clearly understood. If our responses have resolved your concerns, we would greatly appreciate your consideration of a higher score. Thank you again for your time and patience.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.