acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Latent Visual Search for GUI Grounding

10 views

Abstract

GUI grounding is a critical capability for GUI agents powered by vision-language models, helping them execute user instructions by locating the corresponding elements in screenshots. Single-step grounding struggles with small elements and dense layouts, motivating multi-step visual search. However, existing approaches commonly rely on textual reasoning misaligned with visual space or costly multi-round interactions with external visual tools. To make multi-step visual search an explicit spatial process within the model, we propose ***ReLaViS***, which performs ***Re**current **La**tent **Vi**sual **S**earch* in a single interaction round. At each step, a spatial search head uses the hidden state to query the screenshot's visual tokens, producing a spatial search distribution that explicitly represents the search focus. This distribution then aggregates the visual tokens into latent visual evidence, which is recurrently fed back as the next input embedding to condition subsequent search. We further introduce a GUI-aware coarse-to-fine inductive bias through trajectories constructed from flat element annotations, supervising search from the global interface through intermediate element groups to the target. Built on Qwen2.5-VL-7B, ReLaViS improves ScreenSpot-Pro accuracy by 3.1 percentage points to 56.3% with only a 3.5% increase in inference FLOPs and outperforms the matched single-step baseline on all five benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI grounding, vision-language models, latent visual reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PqSBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReLaViS, which formulates GUI grounding as recurrent latent visual search within a single interaction round. At each step, a spatial search head predicts a distribution over visual tokens, which is aggregated into latent visual evidence and fed into the next search step. The model is trained with automatically constructed coarse-to-fine GUI trajectories and achieves 56.3% on ScreenSpot-Pro, outperforming the matched coordinate-free single-step baseline by 3.1 points with 3.5% additional FLOPs.

Critical strengths
  • Explicit spatial search states are interpretable, and the matched baselines, multi-seed evaluation, efficiency analysis, and intervention experiments provide useful evidence that intermediate states affect subsequent predictions.
  • The paper includes ablations on trajectory supervision, recurrent feedback, search steps, training stages, and coordinate readout, together with search dynamics, UI-tree agreement, and qualitative failure cases.
Critical weaknesses
  • The contribution of recurrent visual search is not cleanly isolated. The fixed LVS-SLOT variant already achieves 54.5% (±0.71) versus 56.3% (±0.29) for the full model, suggesting that a substantial portion of the gain may come from coarse-to-fine intermediate supervision rather than recurrent evidence propagation. A key missing baseline is one using the same intermediate trajectory supervision but predicting all intermediate states without recurrent feedback. Moreover, Fig. 4 and RQ3 suggest that early search states have relatively limited influence, which weakens the claimed progressive-search interpretation.

  • The empirical gains and generalization evidence are limited. Improvements over the matched coordinate-free baseline are small on several out-of-domain benchmarks, while the largest gain occurs on the in-domain UI-Vision benchmark. Comparisons with tool-based methods are also not fully controlled because of substantial differences in training data. In addition, the method requires dense flat element annotations for trajectory construction, limiting applicability to datasets with only target annotations.

Questions for authors
  1. Could you provide a baseline using the same GUI-aware intermediate supervision but without recurrent feedback, e.g., parallel auxiliary predictions, to separate the effects of trajectory supervision and recurrence?
  2. Given the limited influence of early search steps in Fig. 4 and RQ3, what information do early ztz_t representations actually contribute? Could you test shorter trajectories such as T=2T=2, or remove s1/s2s_1/s_2 independently?
  3. The 3×3 weighted readout improves substantially over peak-token readout. Do all matched baselines use exactly the same readout, and what is the ReLaViS-vs-single-step improvement under peak-token readout?
5 comments
AuthorsOpenReview ↗

Response to Reviewer PqSB (Part 1/3)

We thank Reviewer PqSB for the careful review and the valuable experimental suggestions. We have conducted the requested experiments and reported the results in the responses below.


W1-1 & Q1: The contribution of recurrent latent visual search is not cleanly isolated.

The fixed LVS-SLOT\texttt{LVS-SLOT} variant removes visual-evidence feedback while retaining cross-step causal attention, so it only partially ablates recurrent latent visual search. ReLaViS’s 1.8 pp gain over this variant (56.3% vs. 54.5%; Table 2) supports the benefit of recurrent visual-evidence feedback.

To cleanly isolate the contribution of recurrent latent visual search, we follow the reviewer’s suggestion and introduce a new parallel auxiliary baseline that uses the same trajectory supervision but no recurrent feedback. Specifically, the parallel auxiliary baseline predicts all sts_t in parallel from the same hidden state using a separate prediction head for each step, following [1].

Table PqSB-1: Accuracy (%) on ScreenSpot-Pro. “Recurrent search” denotes recurrent latent visual search.

VariantTrajectory supervisionRecurrent searchAccuracyΔ\Delta (pp)
Single-step (coordinate-free)✗✗53.2$_{\pm 0.52}$--
Parallel auxiliary (new)✓✗53.4$_{\pm 0.32}$+0.2
Final-only multi-step✗✓54.0$_{\pm 0.48}$+0.8
ReLaViS✓✓56.3$_{\pm 0.29}$+3.1

As shown in Table PqSB-1, our ReLaViS outperforms the parallel auxiliary baseline by 2.9 pp (56.3% vs. 53.4%), supporting the substantial contribution of recurrent latent visual search beyond trajectory supervision alone. Moreover, ReLaViS gains 3.1 pp over the single-step baseline, exceeding the sum of the gains from trajectory supervision alone (0.2 pp) and recurrence alone (0.8 pp), suggesting that the two components play complementary roles in improving grounding.


W1-2 & Q2: What do the early search states actually contribute?

To investigate the contribution of early search states, we remove each of the first two search states (S1S_1 and S2S_2) from the training trajectory, with other configurations unchanged.

Table PqSB-2: Accuracy (%) on ScreenSpot-Pro after removing early search states.

Training trajectoryRemoved stateAccuracyΔ\Delta (pp)
E→C2→C3→e⋆E\to C_2\to C_3\to e^\star (ReLaViS)--56.3$_{\pm 0.29}$--
C2→C3→e⋆C_2\to C_3\to e^\star (new)S1S_155.0$_{\pm 0.26}$-1.3
E→C3→e⋆E\to C_3\to e^\star (new)S2S_254.7$_{\pm 0.63}$-1.6
e⋆e^\star (single-step)all53.2$_{\pm 0.52}$-3.1

As shown in Table PqSB-2, removing the first search state (S1S_1) lowers accuracy by 1.3 pp, and removing the second search state (S2S_2) lowers accuracy by 1.6 pp. These results support the utility of the early search stages, consistent with their intended role in providing global and regional context for subsequent search.

Regarding the limited influence noted by the reviewer, the low target probability mass at early steps in Fig. 4 (Sec. 4.3) is consistent with these states covering broad candidate sets to provide context. Similarly, the small effects at early steps in RQ3 (Sec. 4.2) reflect limited sensitivity to the tested inference-time replacements, whereas removing these steps from the training trajectory lowers accuracy. In addition, with all four steps retained, shuffling the first three reduces accuracy to 53.5% (-2.8 pp; Sec. 4.2, RQ4), supporting the value of the progressive-search order.

AuthorsOpenReview ↗

Response to Reviewer PqSB (Part 2/3)

W2-1: Gains are small on several out-of-domain benchmarks and largest on in-domain UI-Vision.

As shown in Table PqSB-3, although gains are modest on some benchmarks, ReLaViS gains 3.1 and 3.2 pp on ScreenSpot-Pro and OSWorld-G-Refine, respectively, both out-of-domain benchmarks dominated by small targets. These results show that the benefits extend beyond in-domain UI-Vision.

Table PqSB-3: Accuracy gains (pp) of ReLaViS-7B over single-step-7B (coordinate-free), from Table 6 (Appendix C.1). Small targets have a relative area < 0.09%.

BenchmarkSmall-target shareOverall gainSmall-target gain
ScreenSpot-v24%+0.3+0.6
MMBench-GUI-L230%+0.9+1.0
OSWorld-G-Refine59%+3.2+4.7
UI-Vision (in-domain)74%+6.6+7.9
ScreenSpot-Pro80%+3.1+3.3

Moreover, on all five benchmarks, the gain on small targets exceeds the overall gain. This supports the benefit of our ReLaViS for small targets across in-domain and out-of-domain benchmarks, consistent with our focus on small targets in high-resolution, dense interfaces (Sec. 1).


W2-2: Comparisons with tool-based methods are not fully controlled due to differences in training data.

We agree that comparisons with other tool-based methods are not fully controlled: MVP and UI-Zoomer are training-free, while GUI-Spotlight and GUI-Cursor are trained on their own data. We therefore keep their scores in Table 1 as reference results.

For a controlled comparison, we train the single-step baseline and ReLaViS on the same 100K samples, and apply an identical zoom-in procedure (Table 1, L349–350; Appendix B.3, L1151–1152 and L1166–1173). Our ReLaViS (without zoom-in) achieves 68.1% average accuracy in a single interaction round, close to the 68.6% of two-round zoom-in + single-step (Table 1, Sec. 4.1), with roughly half the latency (2.19 vs. 4.28 s/sample on ScreenSpot-Pro; Table 5, Appendix B.4). These results demonstrate that ReLaViS offers a favorable accuracy–efficiency trade-off (Sec. 4.1, L368–394).

When both use an identical zoom-in procedure, ReLaViS improves accuracy by 1.2 pp on ScreenSpot-Pro and 5.0 pp on UI-Vision over the single-step baseline and remains comparable on the other three benchmarks (Table 1). Hence, our ReLaViS can also complement tool-based methods (Sec. 4.1, L368–394).


W2-3: Requiring dense flat element annotations limits applicability to datasets with only target annotations.

To extend applicability to datasets with only target annotations, we construct trajectories using non-target boxes detected by OmniParser v2 [2]. Specifically, we retain GroundCUA’s target boxes and replace its non-target annotations with these detected boxes.

Table PqSB-4: Accuracy (%) on ScreenSpot-Pro.

VariantNon-target element boxesAccuracyGain (pp)
Final-only multi-stepNot used54.0$_{\pm 0.48}$—
ReLaViS w/ detected boxes (new)OmniParser v255.3$_{\pm 0.48}$+1.3
ReLaViS w/ full annotationsGroundCUA56.3$_{\pm 0.29}$+2.3

As shown in Table PqSB-4, ReLaViS trained with detected boxes outperforms the best target-only variant (final-only multi-step) by 1.3 pp (55.3% vs. 54.0%), and trails ReLaViS trained with full annotations (56.3%) by only 1.0 pp. Thus, our ReLaViS can use automatically detected boxes to construct trajectories and improve grounding even on datasets with only target annotations, extending its applicability.

Moreover, as Reviewer ZHUM notes, ReLaViS requires only flat element bounding boxes during training, with no UI trees or inference-time annotations (Sec. 3.2, L210–215; Sec. 3.4, L281–282). Such annotations are available in datasets spanning multiple platforms, including ScaleCUA [3] (cross-platform), GroundCUA [4] (desktop), ScreenParse [5] (web), and RICO [6] (mobile).

AuthorsOpenReview ↗

Response to Reviewer PqSB (Part 3/3)

Q3: Do all matched baselines use the same readout, and what is the ReLaViS-vs-single-step gain under peak-token readout?

Yes. ReLaViS and all matched coordinate-free baselines, including the single-step baseline and ablation variants, use the same readout (3×3 weighted by default; Appendix A.4, L1039–1047, Eq. (13); Appendix B.3, L1148–1151 and L1155–1156). The coordinate-based baseline outputs textual coordinates and uses no readout.

As shown in Table PqSB-5, ReLaViS’s advantage persists under peak-token readout: it outperforms the single-step baseline on all five benchmarks, with an average gain of 2.1 pp (61.1% vs. 59.0%).

Table PqSB-5: Accuracy (%) under two readouts.

ReadoutModelScreenSpot-ProMMBench-GUI-L2OSWorld-G-RefineScreenSpot-v2UI-VisionAverage
Peak-tokenSingle-step44.273.949.989.237.959.0
ReLaViS46.7 (+2.5)74.9 (+1.0)51.4 (+1.5)89.6 (+0.4)42.7 (+4.8)61.1 (+2.1)
3×3 weighted (default)Single-step53.280.259.390.443.165.3
ReLaViS56.3 (+3.1)81.1 (+0.9)62.5 (+3.2)90.7 (+0.3)49.7 (+6.6)68.1 (+2.8)

Unlike peak-token readout, the 3×3 weighted readout mitigates patch-level quantization by allowing clicks between token centers. With this finer readout, ReLaViS’s gains over single-step increase most on the three benchmarks with the highest small-target shares (Table PqSB-3), supporting its advantage in fine-grained localization.


We are revising the paper accordingly and will upload the revised version as soon as possible. We thank Reviewer PqSB again for the helpful suggestions that strengthen our work. If the reviewer finds that the additional experiments and clarifications sufficiently address the main concerns, we would be grateful for a reconsideration of the score.

References

[1] Fabian Gloeckle et al., Better & Faster Large Language Models via Multi-token Prediction. In ICML, 2024.

[2] Yadong Lu et al., OmniParser for Pure Vision Based GUI Agent. arXiv:2408.00203, 2024.

[3] Zhaoyang Liu et al., ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data. In ICLR, 2026.

[4] Aarash Feizi et al., Grounding Computer Use Agents on Human Demonstrations. In ICLR, 2026.

[5] A. Said Gurbuz et al., ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision. In ICML, 2026.

[6] Biplab Deka et al., Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In UIST, 2017.

Reviewer_cHkLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a GUI grounding method that performs recurrent visual search within a single model interaction. At each step, a spatial search head predicts a distribution over screenshot visual tokens; the distribution aggregates visual evidence that is fed into the next step. To supervise this process without annotated search trajectories, the authors construct target-conditioned, coarse-to-fine candidate sets from flat GUI element bounding boxes using geometric partitioning. Training combines intermediate candidate-wise coverage with final-step target coverage and a center-biased distribution loss, using a three-stage schedule. Experiments on five GUI grounding benchmarks show the effectiveness of the proposed method.

Critical strengths

The proposed latent visual search procedure is interesting, and the design of the visual evidence token cleverly avoids the distribution gap.

Critical weaknesses
  1. The candidate-wise coverage loss constrains the visual search distribution at each step to be uniformly distributed across each contained element. However, the right panel of Figure 3 shows that focusing on the ground-truth (GT) element is more beneficial for performance improvement. Would it be more reasonable to design a GT element-centered distribution for supervision here? Even if such supervision is applied at intermediate steps, rather than always requiring the visual weights of contained elements to be uniformly distributed?
  2. Table 1 shows that Single-step-7B (coordinate-free) achieves higher average performance compared to Single-step-7B (coordinate-based). How does the performance of the coordinate-free version of ReLaViS-7B compare? Will it also outperform the coordinate-based version?
  3. The partitioning objective in Section3.2 relies solely on element geometry, and the query decides which branch to retain only through the known target element. For different instructions within the same screenshot, the candidate sets in the first few steps may be identical; geometric nesting also does not necessarily correspond to functional or UI-tree hierarchy. It is suggested to quantify the overlap rate (C2,C3​) across different targets/instructions, and report stratified performance based on the degree of inconsistency between geometric groupings and the actual UI-tree. Then, compare geometric trajectories, ground-truth tree trajectories (on available subsets), or query-aware trajectories.
Questions for authors

none

4 comments
AuthorsOpenReview ↗

Response to Reviewer cHkL (Part 1/2)

We sincerely thank Reviewer cHkL for recognizing our work and for the valuable suggestions. Please find our responses below.


W1: Would GT element-centered intermediate supervision be more beneficial?

To evaluate the suggested GT element-centered intermediate supervision, we compare two variants of the candidate-wise coverage (CWC) loss with its original formulation (Eq. (7), Sec. 3.3, L238–L248): Target-weighted CWC (new addition) raises the coverage lower bound on the GT element to 1/∣Ct∣1/\sqrt{\lvert C_t\rvert}, the geometric mean of the original bound 1/∣Ct∣1/\lvert C_t\rvert and the target-only bound 11, to test moderate GT emphasis. We assign equal coverage lower bounds to the other candidates so that the sum of all lower bounds remains 11. Repeated target (Table 2, RQ1) is the extreme case with a coverage lower bound of 11 on the GT element, supervising the model to focus only on the GT element at every step.

Table cHkL-1: Accuracy (%) on ScreenSpot-Pro under different intermediate supervision variants.

VariantLower bound on targetAccuracyΔ\Delta (pp)
Original CWC (ReLaViS)1/∣Ct∣1/\lvert C_t\rvert56.3$_{\pm 0.29}$--
Target-weighted CWC (new)1/∣Ct∣1/\sqrt{\lvert C_t\rvert}55.2$_{\pm 0.80}$-1.1
Repeated target11 (target only)53.9$_{\pm 0.96}$-2.4

As shown in Table cHkL-1, accuracy drops by 1.1 pp under moderate emphasis on the GT element and by 2.4 pp under target-only supervision. Thus, for intermediate supervision, maintaining coverage over each candidate (CWC) is more beneficial than placing greater emphasis on the GT element.


W2: Does coordinate-free ReLaViS-7B also outperform the coordinate-based version?

Our ReLaViS-7B in Table 1 is coordinate-free by design: it reads the click coordinate from its final-step spatial search distribution without generating coordinate tokens (Sec. 3.4, L278–282). As shown in Table 1, ReLaViS-7B outperforms both single-step-7B (coordinate-based) and single-step-7B (coordinate-free) on all five benchmarks, achieving an average accuracy of 68.1%. This exceeds the coordinate-based baseline (61.6%) by 6.5 pp and the coordinate-free baseline (65.3%) by 2.8 pp.

AuthorsOpenReview ↗

Response to Reviewer cHkL (Part 2/2)

W3: Candidate-set overlap across instructions, stratified performance by UI-tree inconsistency, and trajectory comparisons.

Overlap across different targets/instructions. We enumerate all 154,350 instruction pairs with different targets on the same screenshot. C1=EC_1=E is shared by construction, but the two instructions in each pair yield different C2C_2 sets in 79.7% of pairs (mean set IoU: 0.387) and different C3C_3 sets in 93.7% of pairs (mean set IoU: 0.136). These statistics show that candidate sets already differ by the second step for most instruction pairs.

Stratified performance by UI-tree inconsistency. On Screen2AX-Task [1], we retain 688 instances whose target matches a UI-tree element and stratify them by the mean C2C_2/C3C_3 agreement IoU with the UI tree, computed as in Table 8 (Appendix C.4, L1362–1364).

Table cHkL-2: Accuracy (%) on Screen2AX-Task, stratified by agreement between our trajectories and the UI tree. Lower agreement indicates greater inconsistency.

Agreement with UI treeLow [0, 0.6)Medium [0.6, 0.8)High [0.8, 1]
Number of instances231171286
Single-step-7B (coordinate-free)74.0372.5180.77
ReLaViS-7B75.1872.7181.93
Gain (pp)+1.15+0.20+1.16

As shown in Table cHkL-2, ReLaViS shows a positive gain in each UI-tree agreement tier on Screen2AX-Task, even in the lowest agreement tier.

Comparison with geometric, ground-truth tree, or query-aware trajectories. As Reviewer ZHUM notes, our focus is on constructing effective supervision from flat element annotations. Our GUI-aware trajectories, constructed using only flat element annotations, achieve closer agreement with UI-tree hierarchies and higher grounding accuracy than geometric zoom. Their C2C_2/C3C_3 agreement IoUs are 75.1%/74.7%, compared with 62.4%/57.9% for geometric zoom (Table 8, Appendix C.4). They also improve ScreenSpot-Pro accuracy by 2.5 pp over geometric zoom (56.3% vs. 53.8%; Table 2, RQ4).

For the other two alternatives, desktop grounding training data with aligned UI trees remain scarce (Sec. 3.2, L210–212), and query-aware trajectory construction requires additional design and validation. Therefore, we do not yet compare grounding performance after training with ground-truth tree or query-aware trajectories; future work will investigate these valuable alternatives.


We are revising the paper accordingly and will upload the revised version as soon as possible. We thank Reviewer cHkL again for the helpful suggestions that strengthen our work and hope these responses address the reviewer’s concerns.

References

[1] Viktor Muryn et al., Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation. arXiv:2507.16704, 2025.

Thanks for the awesome reply. They have addressed my concerns and I maintain my initial positive score.

AuthorsOpenReview ↗

Thank you for your kind reply

We’re glad to hear that our response addresses your concerns. We sincerely appreciate the time and care you put into reviewing our paper, and we’re grateful for your thoughtful feedback.

Reviewer_ZHUMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReLaViS, a recurrent visual search method for GUI grounding. It uses spatial distributions to select visual information for subsequent search steps and constructs coarse-to-fine training trajectories from GUI element annotations. Compared with a matched single-step baseline using Qwen2.5-VL-7B, ReLaViS improves ScreenSpot-Pro accuracy from 53.2% to 56.3% and the five-benchmark average from 65.3% to 68.1%. The paper also provides component ablations, spatial-state interventions, efficiency analysis, and experiments with a 3B model.

Critical strengths
  1. ReLaViS combines recurrent visual feedback with GUI-aware intermediate supervision. The spatial distributions guide subsequent search steps and determine the final click location. The training trajectories are constructed from flat GUI element annotations without requiring ground-truth UI trees.

  2. The experiments include a matched single-step baseline using the same backbone, training data, and coordinate readout. Table 2 evaluates different feedback mechanisms, intermediate supervision, and training stages. Figure 3 confirms that intermediate search states affect final predictions. Improvements are also reported with a separately trained 3B model. The additional inference cost is relatively small, with 3.5% more FLOPs and 6.3% higher latency on ScreenSpot-Pro.

Critical weaknesses
  1. GUI-Actor (Wu et al., 2025b) already uses a spatial grounding head, and V2P (Chen et al., 2026b) uses Gaussian localization supervision. ReLaViS adds recurrent visual feedback and GUI-aware intermediate supervision. Although the ablations support the proposed design, the paper could more clearly explain the additional contributions relative to existing methods.

  2. The performance gains vary across benchmarks. The average improvement over four out-of-domain benchmarks is approximately 1.9 percentage points without zoom-in and 0.4 points when both methods use the same zoom-in procedure (Table 1). The largest improvement without zoom-in occurs on the in-domain UI-Vision benchmark. In addition, the main component ablations are limited to ScreenSpot-Pro (Table 2), leaving the benefits across other tasks less established.

Questions for authors
  1. Could the authors clarify the specific contributions of recurrent visual feedback and GUI-aware intermediate supervision compared with GUI-Actor and V2P?
  2. Could the authors evaluate the fixed-slot feedback and final-only supervision variants on another out-of-domain benchmark, such as MMBench-GUI-L2 or OSWorld-G-Refine, to verify whether their benefits extend beyond ScreenSpot-Pro?
  3. In Equation (7), nearby GUI elements may share visual tokens, allowing the same probability mass to satisfy multiple coverage constraints. How does the implementation handle this case, and does it affect supervision for dense GUIs with small targets?
  4. How sensitive is the training trajectory construction to missing annotations of non-target GUI elements? Could the authors clarify the annotation requirements and discuss the impact of incomplete annotations?
  5. Could the authors report how inference latency, FLOPs, and peak memory change with the number of visual tokens, particularly for high-resolution inputs, compared with the matched single-step baseline?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZHUM (Part 1/3)

We thank Reviewer ZHUM for the positive assessment and the constructive suggestions. Please find our responses below.


W1 & Q1: What are the specific contributions of recurrent visual feedback and GUI-aware intermediate supervision compared with GUI-Actor and V2P?

Building on GUI-Actor’s spatial grounding head and V2P’s Gaussian localization supervision, our ReLaViS extends single-step localization to multi-step visual search within one model interaction through two contributions:

  • Recurrent visual feedback uses each spatial search state to aggregate visual evidence and feeds this evidence back to guide subsequent search (Sec. 3.1). Reviewer PqSB highlights the “interpretable” spatial states, while Reviewer cHkL notes that the visual evidence “cleverly avoids the distribution gap.”
  • GUI-aware intermediate supervision extends supervision beyond the final target to the intermediate search process, guiding the search from the global interface through element groups to the target. We construct these trajectories from flat GUI element annotations, without requiring annotated search trajectories or ground-truth UI trees (Secs. 3.2–3.4).

We clarify these contributions relative to GUI-Actor and V2P in Sec. 2 (L107–108) of the revised version, which we will upload shortly.


W2-1: Performance gains vary across benchmarks and with zoom-in, with the largest gain on in-domain UI-Vision.

As shown in Table ZHUM-1, although gains are modest on some benchmarks, ReLaViS gains 3.1 and 3.2 pp on ScreenSpot-Pro and OSWorld-G-Refine, respectively, both out-of-domain benchmarks dominated by small targets. These results show that the benefits extend beyond in-domain UI-Vision.

Table ZHUM-1: Accuracy gains (pp) of ReLaViS-7B over single-step-7B (coordinate-free), from Table 6 (Appendix C.1). Small targets have a relative area < 0.09%.

BenchmarkSmall-target shareOverall gainSmall-target gain
ScreenSpot-v24%+0.3+0.6
MMBench-GUI-L230%+0.9+1.0
OSWorld-G-Refine59%+3.2+4.7
UI-Vision (in-domain)74%+6.6+7.9
ScreenSpot-Pro80%+3.1+3.3

Moreover, on all five benchmarks, the gain on small targets exceeds the overall gain. This supports the benefit of our ReLaViS for small targets across in-domain and out-of-domain benchmarks, consistent with our focus on small targets in high-resolution, dense interfaces (Sec. 1).

The smaller gains with zoom-in are consistent with zoom-in alleviating some of the small-target localization difficulty that ReLaViS addresses. As shown in Table 1 (Sec. 4.1, L349–350), under the same zoom-in procedure, ReLaViS still improves accuracy by 1.2 pp on ScreenSpot-Pro and 5.0 pp on UI-Vision over the matched single-step baseline and remains comparable on the other three benchmarks. These results show that our ReLaViS can also complement thinking with images methods (Sec. 4.1, L368–394).


W2-2 & Q2: How do the fixed-slot feedback and final-only supervision variants perform on another out-of-domain benchmark?

Following the reviewer’s suggestion, we evaluate the fixed-slot feedback and final-only supervision variants on OSWorld-G-Refine, another out-of-domain benchmark (Table ZHUM-2).

Table ZHUM-2: Accuracy (%) of the ablation variants in Table 2 (RQ1 and RQ2).

VariantScreenSpot-ProΔ\Delta (pp)OSWorld-G-Refine (new)Δ\Delta (pp)
ReLaViS56.3$_{\pm 0.29}$--62.5$_{\pm 1.01}$--
Fixed LVS-SLOT\texttt{LVS-SLOT} (w/o recurrent visual feedback)54.5$_{\pm 0.71}$-1.860.8$_{\pm 0.46}$-1.7
Final-only (w/o intermediate supervision)54.0$_{\pm 0.48}$-2.360.6$_{\pm 1.11}$-1.9
Single-step (coordinate-free)53.2$_{\pm 0.52}$-3.159.3$_{\pm 1.18}$-3.2

On OSWorld-G-Refine, removing recurrent visual feedback or GUI-aware intermediate supervision lowers accuracy by 1.7 or 1.9 pp, respectively, consistent with the reductions on ScreenSpot-Pro. These results show that the benefits of both components extend beyond ScreenSpot-Pro.

AuthorsOpenReview ↗

Response to Reviewer ZHUM (Part 2/3)

Q3: How does Eq. (7) handle visual tokens shared by nearby elements, and does this affect supervision for dense GUIs with small targets?

We apply the candidate-wise coverage (CWC) loss (Eq. (7)) directly: a shared token’s probability mass counts toward each candidate whose support contains it. At the final step, only the target remains in the candidate set, so each token’s probability mass contributes solely to the target.

Empirically, CWC remains effective for dense GUIs with small targets without special handling for shared visual tokens: To assess the effect of token sharing, we modify only the intermediate CWC loss to divide each shared token’s probability mass equally among the candidates sharing it, so that the mass is counted only once in total. ReLaViS with CWC achieves 56.3±0.2956.3_{\pm 0.29}% on ScreenSpot-Pro, which is dominated by small targets, compared with 56.1±0.2256.1_{\pm 0.22}% for this variant. Moreover, ReLaViS with CWC achieves larger gains on small targets than overall across all five benchmarks (Table ZHUM-1; Appendix C.1, Table 6).


Q4: How sensitive is training trajectory construction to missing non-target annotations, and what annotations does it require?

ReLaViS requires flat bounding boxes for target and non-target GUI elements to construct training trajectories, with no UI trees or inference-time annotations (Sec. 4, L288–290; Sec. 3.4, L281–282). This requirement can be readily met by existing datasets and tools, as detailed in our response to Reviewer PqSB (W2-3).

To assess sensitivity to missing non-target annotations, we randomly remove 5%–100% of the non-target boxes in GroundCUA while retaining the target boxes. We then compute candidate-set IoU between trajectories constructed from the reduced and full annotations, following Table 8 (Appendix C.4, L1362–1364). As shown in Table ZHUM-3, with 5% removed, the intermediate candidate sets retain IoUs of 0.87 and 0.86 for C2C_2 and C3C_3, respectively; agreement decreases as more annotations are removed.

Table ZHUM-3: Agreement of candidate sets with those constructed from the full GroundCUA annotations and ScreenSpot-Pro accuracy (%).

Non-target boxesC1C_1 IoUC2C_2 IoUC3C_3 IoUAccuracy
Full (GroundCUA)1.001.001.0056.3$_{\pm 0.29}$
5% removed0.960.870.86--
25% removed0.760.560.53--
OmniParser v2$^\ddagger$ (new)0.640.450.4355.3$_{\pm 0.48}$
50% removed0.510.360.35--
100% removed0.020.070.2153.9$_{\pm 0.96}^{\dagger}$

†^\dagger Identical to repeated target (Table 2, RQ1), since only the target box remains.
‡^\ddagger In the OmniParser experiment, original non-target annotations are used only for candidate-set IoU evaluation, with detections matched to annotated elements by center containment.

To assess the impact of incomplete annotations on training, we retain GroundCUA’s target boxes and replace all non-target boxes with those detected by OmniParser v2 [1] from screenshots alone. Training with these trajectories reaches 55.3% on ScreenSpot-Pro, 1.3 pp above the best target-only variant (final-only supervision, 54.0%; Table 2) and only 1.0 pp below training with full annotations (56.3%). These results show that our ReLaViS can use automatically detected boxes to construct trajectories and improve grounding even on incomplete datasets with only target annotations, extending its applicability.

AuthorsOpenReview ↗

Response to Reviewer ZHUM (Part 3/3)

Q5: How do inference latency, FLOPs, and peak memory change with the number of visual tokens compared with the matched single-step baseline?

We report inference latency, FLOPs, and peak memory for ReLaViS and the matched single-step baseline on ScreenSpot-Pro, grouped by the number of visual tokens NN, under the setting of Table 5 (Appendix B.4).

Table ZHUM-4: Inference cost of single-step-7B (coordinate-free) → ReLaViS-7B on ScreenSpot-Pro, grouped by NN.

Visual tokensInstancesLatency (s/sample)Δ\Delta (%)TFLOPs/sampleΔ\Delta (%)Peak memory (GiB)Δ\Delta (GiB)
< 4K1180.97 → 1.06+9.381.21 → 83.59+2.916.40 → 16.63+0.23
4K–6K7501.30 → 1.41+8.5107.46 → 110.78+3.116.67 → 16.97+0.30
6K–8K901.80 → 1.92+6.7151.51 → 156.53+3.317.02 → 17.39+0.37
8K–12K5422.90 → 3.07+5.9247.14 → 256.26+3.717.67 → 18.19+0.52
≥ 12K815.32 → 5.58+4.9451.36 → 470.42+4.218.78 → 19.54+0.76
All1,5812.06 → 2.19+6.3173.51 → 179.65+3.518.78 → 19.54+0.76

As NN increases, the relative latency overhead decreases, while the relative FLOPs overhead and additional peak memory increase moderately. In the N≥12KN\geq12K group, ReLaViS adds 4.9% latency, 4.2% FLOPs, and 0.76 GiB peak memory (approximately 4.0%), keeping all three relative overheads below 5%. These results show that our ReLaViS performs multi-step visual search within one model interaction at an inference cost close to that of the matched single-step baseline, even for high-resolution inputs.


We are revising the paper accordingly and will upload the revised version as soon as possible. We thank Reviewer ZHUM again for the helpful suggestions that strengthen our work and hope these responses address the reviewer’s concerns.

References

[1] Yadong Lu et al., OmniParser for Pure Vision Based GUI Agent. arXiv:2408.00203, 2024.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript: updates in response to reviewers

We sincerely thank all three reviewers for recognizing our work and offering constructive suggestions, and we appreciate Reviewer PqSB’s decision to raise the score. We have revised and uploaded the manuscript accordingly, with changes highlighted in blue. Below, we summarize the updates and their locations in the revised manuscript.


Updates in response to Reviewer ZHUM

W1 & Q1: Contributions relative to GUI-Actor and V2P. We have clarified how ReLaViS extends GUI-Actor and V2P through recurrent latent visual search and GUI-aware intermediate supervision (Sec. 2, L107–109). The method description also explicitly attributes the spatial search head to GUI-Actor (Sec. 3.1, L189–190) and Gaussian target construction to V2P (Sec. 3.3, L256–257).

W2-2 & Q2: Component ablations beyond ScreenSpot-Pro. We have extended the fixed-slot feedback and final-only supervision ablations to OSWorld-G-Refine (Appendix C.3, L1389–1393).

Q3: Shared visual tokens in CWC supervision. We have clarified how CWC counts probability mass when candidates share visual tokens, and added an ablation that divides this mass equally among the candidates (Appendix C.4, L1433–1441).

Q4: Annotation requirements and incomplete annotations. We have clarified the annotation requirements and identified datasets that provide suitable annotations across platforms (Appendix B.1, L1064–1068). To assess the impact of incomplete annotations, we have added an analysis of trajectories with missing non-target boxes and a training experiment with non-target boxes detected by OmniParser v2 (Appendix C.7, L1504–1543).

Q5: Inference cost by visual token count. We have included inference latency, FLOPs, and peak memory measurements for ReLaViS and the matched single-step baseline, grouped by visual token count (Appendix B.4, L1243–1261).


Updates in response to Reviewer cHkL

W1: Target-centered intermediate supervision. We have added a comparison between standard CWC and a target-weighted variant to assess whether stronger emphasis on the target during intermediate supervision improves grounding (Appendix C.4, L1423–1432).

W3: Trajectory overlap and UI-tree agreement. We have added an analysis quantifying the overlap between candidate sets constructed for instructions with different targets on the same screenshot (Appendix B.1, L1122–1127). To examine grounding performance when these trajectories differ from UI-tree hierarchies, we have also included Screen2AX-Task results stratified by trajectory agreement with UI trees (Appendix C.6, L1494–1500). For alternative trajectory supervision, we have identified UI trees and instruction semantics as directions for future work (Sec. 5, L495–497).


Updates in response to Reviewer PqSB

W1-1 & Q1: Isolating the contribution of recurrent search. We have added a parallel auxiliary baseline that uses the same trajectory supervision without recurrent search (Appendix C.3, L1348–1382). This comparison is also summarized in the main text (Sec. 4.2, L414–417).

W1-2 & Q2: Contribution of early search steps. We have added ablations that separately remove the first or second search step from the training trajectory (Appendix C.3, L1383–1388).

W2-3: Applicability to datasets with only target annotations. We have added a training experiment that retains the target annotations and replaces the non-target boxes with OmniParser v2 detections (Appendix C.7, L1510–1543). We have also clarified the annotation requirements and identified datasets that provide suitable annotations across platforms (Appendix B.1, L1064–1068).

Q3: Matched readout comparison. We have added a comparison between ReLaViS and the matched single-step baseline under peak-token and default 3×3 neighborhood readouts across all five benchmarks (Appendix C.5, L1454–1479).


Unless explicitly stated otherwise, section, table, and line references in our earlier responses refer to the original submission.

We sincerely thank all reviewers again for their time and deeply appreciate their efforts to help us strengthen our work.

AuthorsOpenReview ↗

Discussion summary

We sincerely thank the Area Chair for considering our paper and responses. We also thank all three reviewers for their careful reading, constructive suggestions, and recognition of our work.

We are glad that our responses address the reviewers’ main concerns. Following Reviewer PqSB’s decision to raise their rating from 2 to 3, all three reviewers now give the paper a positive rating of 3.

Core contributions

ReLaViS performs explicit multi-step spatial search for GUI grounding in a single model interaction without external tools. At each step, ReLaViS recurrently updates a search state aligned with visual tokens and then uses it to aggregate latent visual evidence that guides the next step. Our GUI-aware trajectories, constructed from flat element annotations without requiring UI trees, supervise the search from the global interface through element groups to the target.

Reviewer cHkL finds our recurrent latent visual search “interesting” and notes that the visual evidence “cleverly avoids the distribution gap.” Reviewer PqSB describes the search states as “interpretable” and cites our intervention experiments as evidence that these states affect grounding. Reviewer ZHUM highlights the combination of recurrent latent visual search and GUI-aware intermediate supervision.

ReLaViS outperforms the matched single-step baseline on all five benchmarks. On ScreenSpot-Pro, it improves accuracy from 53.2% to 56.3% with only 3.5% more inference FLOPs. Reviewer ZHUM describes this inference overhead as “relatively small.” These results suggest that our recurrent latent visual search is a valuable direction for GUI grounding, particularly for small targets in high-resolution interfaces.

Discussion summary

We have uploaded the revised paper and listed the changes in a separate comment. Below, we summarize our responses to each reviewer’s main concerns.


Response to Reviewer ZHUM

W2-1: Gains vary across benchmarks (also PqSB W2-1). ReLaViS outperforms the single-step baseline by over 3 pp on both ScreenSpot-Pro and OSWorld-G-Refine, two out-of-domain benchmarks dominated by small targets. Across all five benchmarks, gains on small targets exceed overall gains, consistent with our focus on small targets in high-resolution interfaces.

W2-2 & Q2: Component ablations beyond ScreenSpot-Pro. On OSWorld-G-Refine, ablating recurrent visual feedback or intermediate supervision lowers accuracy by 1.7 or 1.9 pp, respectively. The benefits of both components therefore extend beyond ScreenSpot-Pro.

Q4: Missing non-target annotations (also PqSB W2-3). With non-target boxes from OmniParser v2, ReLaViS outperforms the best target-only variant (final-only supervision) by 1.3 pp on ScreenSpot-Pro. ReLaViS can thus use automatically detected boxes to improve grounding, extending its applicability to datasets with only target annotations.


Response to Reviewer cHkL

W1: Target emphasis during intermediate search. On ScreenSpot-Pro, both target-weighted and target-only supervision underperform our candidate-wise coverage. Thus, maintaining coverage over each candidate is more effective than placing greater emphasis on the target.

W3: Candidate-set overlap and agreement with UI trees. For most pairs of instructions with different targets on the same screenshot, the candidate sets already differ by the second step. Compared with geometric zoom, our trajectories also agree more closely with UI trees and yield higher grounding accuracy. On Screen2AX-Task, ReLaViS improves on the single-step baseline at each level of UI-tree agreement, even the lowest.


Response to Reviewer PqSB

W1-1 & Q1: Isolating the contribution of recurrent latent visual search. With identical trajectory supervision, ReLaViS outperforms the parallel auxiliary baseline without recurrent latent visual search by 2.9 pp on ScreenSpot-Pro. Recurrent search therefore improves grounding beyond trajectory supervision alone, and the ablations suggest that the two components play complementary roles.

W1-2 & Q2: Contribution of the early search steps. Retraining without the first or second step lowers ScreenSpot-Pro accuracy by 1.3 or 1.6 pp, respectively. Thus, both early steps aid subsequent search, consistent with their intended roles in providing global and regional context.

W2-2: Comparison with tool-based methods (also ZHUM W2-1). Single-round ReLaViS approaches the two-round zoom-in baseline in five-benchmark average accuracy (68.1% vs. 68.6%) at roughly half its ScreenSpot-Pro latency. When both use identical zoom-in, ReLaViS also outperforms the baseline on ScreenSpot-Pro and UI-Vision. ReLaViS thus offers a favorable accuracy–efficiency trade-off and can complement tool-based methods.


We thank the Area Chair and all three reviewers again for their valuable time and for helping us strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.