- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReLaViS, which formulates GUI grounding as recurrent latent visual search within a single interaction round. At each step, a spatial search head predicts a distribution over visual tokens, which is aggregated into latent visual evidence and fed into the next search step. The model is trained with automatically constructed coarse-to-fine GUI trajectories and achieves 56.3% on ScreenSpot-Pro, outperforming the matched coordinate-free single-step baseline by 3.1 points with 3.5% additional FLOPs.
Critical strengths
- Explicit spatial search states are interpretable, and the matched baselines, multi-seed evaluation, efficiency analysis, and intervention experiments provide useful evidence that intermediate states affect subsequent predictions.
- The paper includes ablations on trajectory supervision, recurrent feedback, search steps, training stages, and coordinate readout, together with search dynamics, UI-tree agreement, and qualitative failure cases.
Critical weaknesses
-
The contribution of recurrent visual search is not cleanly isolated. The fixed LVS-SLOT variant already achieves 54.5% (±0.71) versus 56.3% (±0.29) for the full model, suggesting that a substantial portion of the gain may come from coarse-to-fine intermediate supervision rather than recurrent evidence propagation. A key missing baseline is one using the same intermediate trajectory supervision but predicting all intermediate states without recurrent feedback. Moreover, Fig. 4 and RQ3 suggest that early search states have relatively limited influence, which weakens the claimed progressive-search interpretation.
-
The empirical gains and generalization evidence are limited. Improvements over the matched coordinate-free baseline are small on several out-of-domain benchmarks, while the largest gain occurs on the in-domain UI-Vision benchmark. Comparisons with tool-based methods are also not fully controlled because of substantial differences in training data. In addition, the method requires dense flat element annotations for trajectory construction, limiting applicability to datasets with only target annotations.
Questions for authors
- Could you provide a baseline using the same GUI-aware intermediate supervision but without recurrent feedback, e.g., parallel auxiliary predictions, to separate the effects of trajectory supervision and recurrence?
- Given the limited influence of early search steps in Fig. 4 and RQ3, what information do early representations actually contribute? Could you test shorter trajectories such as , or remove independently?
- The 3×3 weighted readout improves substantially over peak-token readout. Do all matched baselines use exactly the same readout, and what is the ReLaViS-vs-single-step improvement under peak-token readout?
5 comments
Response to Reviewer PqSB (Part 2/3)
W2-1: Gains are small on several out-of-domain benchmarks and largest on in-domain UI-Vision.
As shown in Table PqSB-3, although gains are modest on some benchmarks, ReLaViS gains 3.1 and 3.2 pp on ScreenSpot-Pro and OSWorld-G-Refine, respectively, both out-of-domain benchmarks dominated by small targets. These results show that the benefits extend beyond in-domain UI-Vision.
Table PqSB-3: Accuracy gains (pp) of ReLaViS-7B over single-step-7B (coordinate-free), from Table 6 (Appendix C.1). Small targets have a relative area < 0.09%.
| Benchmark | Small-target share | Overall gain | Small-target gain |
|---|---|---|---|
| ScreenSpot-v2 | 4% | +0.3 | +0.6 |
| MMBench-GUI-L2 | 30% | +0.9 | +1.0 |
| OSWorld-G-Refine | 59% | +3.2 | +4.7 |
| UI-Vision (in-domain) | 74% | +6.6 | +7.9 |
| ScreenSpot-Pro | 80% | +3.1 | +3.3 |
Moreover, on all five benchmarks, the gain on small targets exceeds the overall gain. This supports the benefit of our ReLaViS for small targets across in-domain and out-of-domain benchmarks, consistent with our focus on small targets in high-resolution, dense interfaces (Sec. 1).
W2-2: Comparisons with tool-based methods are not fully controlled due to differences in training data.
We agree that comparisons with other tool-based methods are not fully controlled: MVP and UI-Zoomer are training-free, while GUI-Spotlight and GUI-Cursor are trained on their own data. We therefore keep their scores in Table 1 as reference results.
For a controlled comparison, we train the single-step baseline and ReLaViS on the same 100K samples, and apply an identical zoom-in procedure (Table 1, L349–350; Appendix B.3, L1151–1152 and L1166–1173). Our ReLaViS (without zoom-in) achieves 68.1% average accuracy in a single interaction round, close to the 68.6% of two-round zoom-in + single-step (Table 1, Sec. 4.1), with roughly half the latency (2.19 vs. 4.28 s/sample on ScreenSpot-Pro; Table 5, Appendix B.4). These results demonstrate that ReLaViS offers a favorable accuracy–efficiency trade-off (Sec. 4.1, L368–394).
When both use an identical zoom-in procedure, ReLaViS improves accuracy by 1.2 pp on ScreenSpot-Pro and 5.0 pp on UI-Vision over the single-step baseline and remains comparable on the other three benchmarks (Table 1). Hence, our ReLaViS can also complement tool-based methods (Sec. 4.1, L368–394).
W2-3: Requiring dense flat element annotations limits applicability to datasets with only target annotations.
To extend applicability to datasets with only target annotations, we construct trajectories using non-target boxes detected by OmniParser v2 [2]. Specifically, we retain GroundCUA’s target boxes and replace its non-target annotations with these detected boxes.
Table PqSB-4: Accuracy (%) on ScreenSpot-Pro.
| Variant | Non-target element boxes | Accuracy | Gain (pp) |
|---|---|---|---|
| Final-only multi-step | Not used | 54.0$_{\pm 0.48}$ | — |
| ReLaViS w/ detected boxes (new) | OmniParser v2 | 55.3$_{\pm 0.48}$ | +1.3 |
| ReLaViS w/ full annotations | GroundCUA | 56.3$_{\pm 0.29}$ | +2.3 |
As shown in Table PqSB-4, ReLaViS trained with detected boxes outperforms the best target-only variant (final-only multi-step) by 1.3 pp (55.3% vs. 54.0%), and trails ReLaViS trained with full annotations (56.3%) by only 1.0 pp. Thus, our ReLaViS can use automatically detected boxes to construct trajectories and improve grounding even on datasets with only target annotations, extending its applicability.
Moreover, as Reviewer ZHUM notes, ReLaViS requires only flat element bounding boxes during training, with no UI trees or inference-time annotations (Sec. 3.2, L210–215; Sec. 3.4, L281–282). Such annotations are available in datasets spanning multiple platforms, including ScaleCUA [3] (cross-platform), GroundCUA [4] (desktop), ScreenParse [5] (web), and RICO [6] (mobile).
Response to Reviewer PqSB (Part 3/3)
Q3: Do all matched baselines use the same readout, and what is the ReLaViS-vs-single-step gain under peak-token readout?
Yes. ReLaViS and all matched coordinate-free baselines, including the single-step baseline and ablation variants, use the same readout (3×3 weighted by default; Appendix A.4, L1039–1047, Eq. (13); Appendix B.3, L1148–1151 and L1155–1156). The coordinate-based baseline outputs textual coordinates and uses no readout.
As shown in Table PqSB-5, ReLaViS’s advantage persists under peak-token readout: it outperforms the single-step baseline on all five benchmarks, with an average gain of 2.1 pp (61.1% vs. 59.0%).
Table PqSB-5: Accuracy (%) under two readouts.
| Readout | Model | ScreenSpot-Pro | MMBench-GUI-L2 | OSWorld-G-Refine | ScreenSpot-v2 | UI-Vision | Average |
|---|---|---|---|---|---|---|---|
| Peak-token | Single-step | 44.2 | 73.9 | 49.9 | 89.2 | 37.9 | 59.0 |
| ReLaViS | 46.7 (+2.5) | 74.9 (+1.0) | 51.4 (+1.5) | 89.6 (+0.4) | 42.7 (+4.8) | 61.1 (+2.1) | |
| 3×3 weighted (default) | Single-step | 53.2 | 80.2 | 59.3 | 90.4 | 43.1 | 65.3 |
| ReLaViS | 56.3 (+3.1) | 81.1 (+0.9) | 62.5 (+3.2) | 90.7 (+0.3) | 49.7 (+6.6) | 68.1 (+2.8) |
Unlike peak-token readout, the 3×3 weighted readout mitigates patch-level quantization by allowing clicks between token centers. With this finer readout, ReLaViS’s gains over single-step increase most on the three benchmarks with the highest small-target shares (Table PqSB-3), supporting its advantage in fine-grained localization.
We are revising the paper accordingly and will upload the revised version as soon as possible. We thank Reviewer PqSB again for the helpful suggestions that strengthen our work. If the reviewer finds that the additional experiments and clarifications sufficiently address the main concerns, we would be grateful for a reconsideration of the score.
References
[1] Fabian Gloeckle et al., Better & Faster Large Language Models via Multi-token Prediction. In ICML, 2024.
[2] Yadong Lu et al., OmniParser for Pure Vision Based GUI Agent. arXiv:2408.00203, 2024.
[3] Zhaoyang Liu et al., ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data. In ICLR, 2026.
[4] Aarash Feizi et al., Grounding Computer Use Agents on Human Demonstrations. In ICLR, 2026.
[5] A. Said Gurbuz et al., ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision. In ICML, 2026.
[6] Biplab Deka et al., Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In UIST, 2017.
Response to Reviewer PqSB (Part 1/3)
We thank Reviewer PqSB for the careful review and the valuable experimental suggestions. We have conducted the requested experiments and reported the results in the responses below.
W1-1 & Q1: The contribution of recurrent latent visual search is not cleanly isolated.
The fixed LVS-SLOT variant removes visual-evidence feedback while retaining cross-step causal attention, so it only partially ablates recurrent latent visual search. ReLaViS’s 1.8 pp gain over this variant (56.3% vs. 54.5%; Table 2) supports the benefit of recurrent visual-evidence feedback.
To cleanly isolate the contribution of recurrent latent visual search, we follow the reviewer’s suggestion and introduce a new parallel auxiliary baseline that uses the same trajectory supervision but no recurrent feedback. Specifically, the parallel auxiliary baseline predicts all st in parallel from the same hidden state using a separate prediction head for each step, following [1].
Table PqSB-1: Accuracy (%) on ScreenSpot-Pro. “Recurrent search” denotes recurrent latent visual search.
As shown in Table PqSB-1, our ReLaViS outperforms the parallel auxiliary baseline by 2.9 pp (56.3% vs. 53.4%), supporting the substantial contribution of recurrent latent visual search beyond trajectory supervision alone. Moreover, ReLaViS gains 3.1 pp over the single-step baseline, exceeding the sum of the gains from trajectory supervision alone (0.2 pp) and recurrence alone (0.8 pp), suggesting that the two components play complementary roles in improving grounding.
W1-2 & Q2: What do the early search states actually contribute?
To investigate the contribution of early search states, we remove each of the first two search states (S1 and S2) from the training trajectory, with other configurations unchanged.
Table PqSB-2: Accuracy (%) on ScreenSpot-Pro after removing early search states.
As shown in Table PqSB-2, removing the first search state (S1) lowers accuracy by 1.3 pp, and removing the second search state (S2) lowers accuracy by 1.6 pp. These results support the utility of the early search stages, consistent with their intended role in providing global and regional context for subsequent search.
Regarding the limited influence noted by the reviewer, the low target probability mass at early steps in Fig. 4 (Sec. 4.3) is consistent with these states covering broad candidate sets to provide context. Similarly, the small effects at early steps in RQ3 (Sec. 4.2) reflect limited sensitivity to the tested inference-time replacements, whereas removing these steps from the training trajectory lowers accuracy. In addition, with all four steps retained, shuffling the first three reduces accuracy to 53.5% (-2.8 pp; Sec. 4.2, RQ4), supporting the value of the progressive-search order.