- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses the limitations of binary rewards in GUI grounding. It proposes GSVPO, which uses entropy-guided local re-observation and agreement across trajectories to produce a continuous, label-free reward. The authors evaluate it for training-free inference, test-time reinforcement learning, and reward augmentation.
Critical strengths
-
A scalable label-free training signal. Building on the idea of test-time reinforcement learning, the paper turns the model’s approximate localization ability into a graded reward without ground-truth coordinates. This could reduce reliance on costly GUI target annotations as training data scales.
-
Broad empirical gains. The method improves grounding across several GUI-specialized models and benchmarks, with particularly clear gains on challenging interfaces. This suggests the approach is useful beyond a single model architecture.
Critical weaknesses
The contribution of cross-trajectory verification is not isolated. In Table 3, selecting only zoomed-in clicks achieves 33.1% on ScreenSpot-Pro, compared with 33.2% for full GSVPO. Without a matched comparison that removes the consensus reward while retaining local re-observation, it is unclear whether the paper’s central mechanism drives the inference gains. The missing latency and cost measurements further limit the practical assessment of this test-time method.
The evaluation does not adequately establish the advantage over the closest alternatives. The paper does not directly compare its label-free adaptation with Learning from Reliable Negatives, or its response to binary-reward saturation with online filtering such as DAPO. It also does not clearly distinguish its motivation from prior dense, ground-truth-based rewards. These gaps make it difficult to assess the novelty and necessity of GSVPO relative to existing approaches.
Questions for authors
Introduction
- The paper states that most prior methods use binary click rewards, but SE-GUI (NeurIPS 2025) and GUI-G² (AAAI 2026) propose distance-based and Gaussian dense rewards. How does GSVPO’s motivation differ from these approaches?
- The paper argues that static data filtering becomes ineffective as the policy changes. Could online filtering, as used in DAPO, address this issue?
- Is the claim about how humans inspect and verify difficult clicks supported by empirical research?
- What evidence supports using coordinate-token entropy to determine crop size across targets of different sizes?
Related work
- The label-free grounding discussion omits the closely related ECCV 2026 paper [1]. Please explain how GSVPO differs from it.
Presentation
- Figure 2 does not appear to be cited in the text. Its labels “Q” and “S” are also unclear, since the method defines the screenshot–instruction pair as .
Experiments
- As a test-time scaling method, GSVPO needs a systematic best-of- direct-sampling baseline under a matched computation budget.
- Please report inference latency and cost. GSVPO generates multiple full-screen proposals and performs a local prediction for each, but the paper does not clearly quantify the added model calls or accuracy–cost trade-off.
[1] Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding.
4 comments
W2. Advantage, difference from other methods. and Q1.Introduction 2 (Difference from DAPO). First, our evaluation demonstrates that magnification-based self-verification yields significant gains on the more challenging ScreenSpot-Pro, and our method outperforms comparable methods such as GUI-RC, BAMI, and UI-Zoomer. Second, in test-time RL, our method achieves better generalization on unseen data, outperforming R-OPSD on ScreenSpot-Pro, OSWorld-G, and OSWorld-G-R, and surpassing GUI-RCPO in the vast majority of comparison cases, which sufficiently demonstrates our performance advantage. Second, our method aims to obtain richer training information without ground truth through self-verification. Through careful search and reading, we consider GUI-RC, GUI-RCPO, and R-OPSD to be methods of the same type as ours, so we discussed them and conducted experimental comparisons, which is sufficient. The paper "Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding" was released on September 14, 2026, only four days before the ICLR 2027 abstract deadline; using this paper to argue that our comparison is insufficient would be unreasonable. Our search and comparison are sufficiently thorough, and regarding DAPO, the overlap with DAPO lies in the motivation, not the mechanism. DAPO retains binary rewards, discards all-correct or all-wrong groups, and resamples until enough informative groups are obtained; discarded rollouts provide no gradient despite their generation cost. GSVPO instead retains these groups and derives a continuous, label-free advantage from cross-trajectory spatial agreement, extracting relative preferences even when all binary rewards are identical. DAPO is therefore a batch-sampling strategy requiring an external correctness signal, whereas GSVPO is a reward-construction mechanism and is orthogonal to DAPO. To prevent similar confusion among future readers, we will add a discussion of the differences in the appendix. Finally, you believe that we have not distinguished our method from other types of ground-truth-based rewards, but our primary starting point is the label-free reward setting, where the reward comes from the model's own magnification-based verification. This starting point is well-grounded (supported by the analytical experiments at the beginning of the paper) and is novel in the area; the inspiration for cross-trajectory self-verification comes from human intuition when identifying complex click targets, as well as from the phenomena discovered and conjectures formed in our earlier analytical experiments.
Q1. Introduction 1 (Difference from SE-GUI, GUI-G2). Thank you for this important distinction. The key difference is the source of supervision. SE-GUI computes distance to the annotated target, while GUI-G² centers its Gaussian point and coverage rewards on the ground-truth element. Their dense rewards therefore still require target coordinates or bounding boxes. Moreover, we have never stated in the paper that these frontier methods rely solely on binary rewards. What we said is that although some methods have proposed different rewards, most of them still need binary rewards to provide training signals to varying degrees. In these methods, hit and miss is an important supervisory signal, and many methods rely on ground-truth labels. This is where GSVPO differs from them. GSVPO does not use the ground-truth target. It estimates spatial preference from agreement among independently cropped and mapped-back local predictions. This enables the same reward to support label-free test-time training and frozen-policy candidate selection, where ground-truth-based rewards cannot be computed. Thus, our contribution is not the Gaussian kernel itself, but label-free cross-trajectory local self-verification as the source of the dense reward.
Q1. Introduction 3 (whether empirical research supports human behavior). We mention the ways humans might approach complex problems only as a starting point for the genesis of our idea. We drew inspiration from this possible human behavior and then developed our method. The strong basis for our method lies in the phenomena we observed in Fig.1(a) and (b), from which the idea emerged, and we subsequently validated it with three RQs and experiments on six benchmarks. Our experiments are sufficient. We never claim in the paper that the effectiveness of our method depends on this human behavior; it is merely a statement to help readers better understand our idea and where it came from.
Q1. Introduction 4 (entropy to crop size). Thank you for this question. We use coordinate-token entropy as a model-intrinsic measure of localization uncertainty, rather than as a direct estimator of physical target size. This interpretation is supported by recent studies. Li et al. [1] show that high-entropy tokens in visual grounding are concentrated on predicted coordinates. Luo et al. [2] further demonstrate that uncertainty computed from generated-token distributions correlates with the preferred visual resolution, while Gröpl et al. [3] use next-token entropy gradients to retrieve visual evidence and guide iterative zooming. Our specific entropy-to-crop mapping is validated empirically. On ScreenSpot-Pro, a 0.3 x 0.3 crop improves over 0.5 x 0.5 by 1.9 points, whereas 0.7 x 0.7 reduces accuracy by 4.8 points. At fixed area, extending the crop along the higher-entropy coordinate axis improves accuracy by 3.2 points, while compressing that axis reduces it by 1.4 points. Every design mentioned in our method is supported by experiments. The same holds for this design, and controlled-variable experiments demonstrate its effectiveness.
[1] The Role of Entropy in Visual Grounding: Analysis and Optimization, Findings of EMNLP 2026
[2] Task-Aware Resolution Optimization for Visual Large Language Models, EMNLP 2025
[3] Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models, ECCV 2026
Q2. Related Work (Difference from an ECCV work). Thank you for pointing out this concurrent work. [1] selects the full-image prediction with the highest coordinate-token confidence as a single pseudo-label, assigns binary rewards by distance to this anchor, and updates only reliable negative samples. GSVPO uses coordinate entropy only to determine the local observation scale; it re-observes every proposal, maps all refined clicks back to the original image, and derives a continuous reward from cross-trajectory self-verified agreement. Thus, GSVPO does not trust any single prediction as a pseudo-label; it attempts to break stubborn errors and error propagation across trajectories and to obtain fine-grained information through magnification-based self-verification. It additionally supports training-free reweighting, label-free adaptation, and reward augmentation. Therefore, I think they are different in the starting point of the method design, the specific sub-problem they try to solve, the design of the reward source, and the definition of the problem, and there is even a certain gap between them. In contrast, GUI-RC, GUI-RCPO, and R-OPSD, which are repeatedly mentioned in the paper, are more worth discussing and are works that walk on a similar research path. Importantly, [1] first appeared on arXiv on September 14, 2026, only four days before the September 18 ICLR 2027 abstract deadline. It was therefore concurrent work that was not reasonably available for a full comparison during submission, and its absence should not be interpreted as overlooking established prior work.
We will cite it and clarify these distinctions in the revision.
[1] Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding.
Q3. Presentation. Thank you for noting this presentation issue. In Fig. 2, “Q” denotes the user query/instruction and “S” denotes the agent’s visual state, i.e., the current screenshot. We originally considered this query–state notation to be common in agent literature and therefore treated it as a convention in the field, but to prevent similar confusion, we will revise it accordingly. We will replace “S” and “Q” with “I” and “q”, respectively, and explicitly cite Fig. 2 in the method overview.
Q4. Experiments 1 (Best-of-(N)). We thank the reviewer for suggesting this control. We compare GSVPO with direct best-of-N sampling using the same model, sampling configuration, examples, and total number of generations. Direct BoN samples exclusively from the full screenshot and selects by native sequence score, whereas GSVPO divides the same budget between full-image proposals and paired local re-observations. We additionally report Direct oracle as a non-deployable upper bound on full-image candidate coverage.
| Method | N=1 | N=2 | N=4 | N=8 | N=18 |
|---|---|---|---|---|---|
| Direct native-score BoN | 13.0 | 13.4 | 13.8 | 13.8 | 15.0 |
| Direct oracle | 13.0 | 14.2 | 17.4 | 22.0 | 27.0 |
| GSVPO | - | 23.6 | 26.8 | 29.0 | 33.2 |
These compute-matched results show that the improvement cannot be explained by additional sampling alone. Local re-observation produces better candidates, while cross-trajectory verification selects them effectively.
Thank you very much for your patience with our work and for your valuable suggestions. We will address your concerns one by one below.
W1. Cross-trajectory verification is not isolated and Q4. Experiment 2 (latency and cost). Thank you for this suggestion. We agree that the previous zoom-only row retained the cross-trajectory reward. However, in our "Single-traj zoom-in click" and "Single-traj full-screen click" experimental designs, cross-trajectory consensus is directly isolated. These two experiments are designed to consider only each trajectory's individual local re-observation under the same settings as GSVPO, without considering the consensus among multiple samples brought by cross-trajectory information. Their final accuracies are 26.1% and 19.3%, respectively. We add a matched ablation using exactly the same global proposals, crops, local re-observations, and zoom-candidate output space, while setting the self-verification advantage to zero. Selecting zoom candidates only by their native local sequence scores obtains 27.6%, whereas adding the cross-trajectory advantage obtains 33.2% (+5.6 points). A single greedy click followed by one local re-click obtains 26.7%, which is still lower than the GSVPO result. Therefore, whether considering the performance gains from local re-observation under high-temperature sampling or from local re-observation under greedy clicking, the comparison with GSVPO demonstrates the effective improvement brought by the cross-trajectory reward. These results separate the gain from magnification itself from the additional gain provided by cross-trajectory verification. We measured latency and computational cost on the same fixed ScreenSpot-Pro samples. Each sample was processed on one RTX 4090; multiple GPUs only parallelized the benchmark. Latency includes preprocessing, generation, decoding/scoring, and selection, while excluding model loading, warm-up, and disk I/O.
*The four-trajectory setting uses eight candidate generations plus one entropy-estimation call.
P90 latency denotes the 90th-percentile per-example latency: 90% of the evaluated examples finish within this time.
For full GSVPO, cross-trajectory reward computation and candidate selection require only 0.135 ms per sample; nearly all additional latency comes from the VLM generations. Its average peak allocated GPU memory is 8.45 GiB. Therefore, Table 3 variants that reuse the same 18 candidates have identical neural inference cost and differ only in negligible selection overhead. We will add these measurements and explicitly discuss this accuracy-efficiency trade-off.