acceptodds
Withdrawn from ICLR 2027

Seeing the Forest and the Trees: Multi-Scale Voting Improves Test-Time GUI Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

GUI grounding, the localization of screen elements based on natural language instructions, is essential for autonomous GUI agents. Supervised training achieves strong accuracy but requires massive labeled data, making annotation expensive and limiting generalization to new environments. Recent work addresses this challenge by exploring test-time grounding, which exploits unlabeled data available during inference via iterative cropping and spatial voting. In this paper, we identify scale variation as an important yet overlooked factor in test-time GUI grounding and propose a simple Multiscale Spatial Voting (MSV) method. Specifically, MSV scales the full image up and down to multiple resolutions, generates multiple predictions at each scale, and aggregates these predictions via spatial voting to produce the final output. We integrate MSV into various test-time grounding approaches, including three training-free methods and two training-based methods. Extensive experiments on four datasets with various backbones confirm that MSV consistently improves grounding performance. Despite its simplicity, we believe this work provides a promising path toward label-efficient GUI agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI grounding, data augmentation, test-time

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eqmLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multiscale Spatial Voting (MSV) for test-time GUI grounding. MSV resizes the full screenshot to multiple scales, performs grounding at each scale, maps predictions back to the original coordinates, and aggregates them using spatial voting. The method is evaluated on multiple GUI benchmarks and backbones, showing improvements in many settings.

Critical strengths

The paper is clearly written, and the method is straightforward and easy to follow.

Critical weaknesses

The methodological novelty is limited. While I am generally cautious about evaluating a paper primarily based on novelty, multi-scale image resizing, training, and inference have been extensively studied in object detection for many years (YOLOv2 [1], SNIP [2], Autofocus [3], etc.). Here, the paper mainly uses this classical technique as an augmentation strategy for existing GUI grounding methods, without introducing a new training method or learning framework.

The experiments do not validate the claimed mechanism. The motivation of MSV is to better handle GUI elements of different sizes, especially tiny icons (Line 71~75). However, the paper only reports aggregate grounding accuracy and does not analyze performance according to target size or relative bounding-box area. Therefore, it is unclear whether the gains actually come from better handling of targets at different scales, rather than simply from increased inference computation.

reference:

[1] Redmon J, Farhadi A. YOLO9000: better, faster, stronger[C]//2017 IEEE conference on computer vision and pattern recognition (CVPR). Ieee, 2017: 6517-6525.

[2] Singh B, Davis L S. An analysis of scale invariance in object detection-SNIP[C]//2018 IEEE/CVF Conference on computer vision and pattern recognition. IEEE, 2018: 3578-3587.

[3] Najibi M, Singh B, Davis L S. Autofocus: Efficient multi-scale inference[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 9745-9755.

Questions for authors
  1. Can the authors report performance separately for tiny/small/medium/large GUI elements to directly validate the main motivation of MSV?

  2. Given that multi-scale resizing and inference are classical techniques in computer vision, what is the key technical contribution of MSV beyond applying this established strategy to existing GUI grounding methods?

Reviewer_QGfjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Multi-Scale Spatial Voting (MSV) for test-time GUI grounding. Instead of cropping or sampling at a single scale, MSV resizes the full screenshot to multiple scales, maps predictions back to the original image, and aggregates them via spatial voting. The authors plug MSV into four existing test-time methods and show consistent gains across four grounding benchmarks.

Critical strengths
  1. Very simple and easy to plug into existing training-free or TTRL methods without changing the model.
  2. Resizing the whole image keeps global layout context, which works noticeably better than random cropping (Table 5), especially on high-res interfaces like ScreenSpot-Pro (Table 3).
  3. Fair rollout budget comparison (fixing K=64K=64 total samples across scales) and clean ablations on scale factors and hyperparameters.
Critical weaknesses
  1. Multi-scale test-time augmentation (TTA) with inverse coordinate mapping is a standard vision trick. Combining it with GUI-RC's voting grid works well, but the technical novelty is thin.
  2. Fixing K does not mean equal compute. Upscaling 4K images on ScreenSpot-Pro significantly increases visual tokens and latency in Qwen2.5-VL, could you please report related cost?
  3. MSV hurts MVP on OSWorld-G with GUI-G2-7B, which isn't really explained.
  4. In GUI-RCPO, the paper does not disentangle the gain of using multi-scale rewards during RL training from simply applying multi-scale voting at inference time.
Questions for authors

See weakness.

Reviewer_ycoyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors focus on improving the accuracy of GUI grounding through test-time compute scaling. They introduce a simple multi-scale voting algorithm for grounding and explore how it performs when combined with training-free grounding methods (or test-time RL). Experiments are conducted on GUI grounding benchmark datasets, mainly using Qwen2.5-VL-3B/7B and its fine-tuned variant.

Critical strengths
  • The main advantage of the proposed method is that it does not require retraining the model and can be seamlessly integrated with other relevant works; however, the new idea or innovation it brings to the community seems somewhat limited.
  • Across the four GUI grounding benchmarks, the improvements appear quite consistent regardless of which test-time method is combined with.
Critical weaknesses
  • Scaling and zoom-in approaches to GUI grounding are already well-known and have been extensively studied. Beyond works the authors already cite, such as RegionFocus and DiMo-GUI, there are a number of works the authors appear to have missed - e.g., R-VLM (ACL Findings 2025) and BAMI (CVPR 2026) - which consistently show that scaling images helps GUI agents; R-VLM zooms in according to the estimated target region, and BAMI further refines the zoom-in policy. What I want to point out is that voting over multi-scale images to improve grounding accuracy is too incremental given the current literature. At a minimum, the authors should explore more promising, underexplored directions: Do we always need voting? What about easy grounding cases? If GUI agents are serving every query in long-horizon tasks, is voting necessary at every step? What about triggering multi-scale inference only when an action appears to fail?

These are just a few examples, but simply applying multi-scale voting without any real advance to the scaling approach itself is marginal.

  • The analysis of wall-clock time and FLOPs when combining the method with existing approaches is also lacking. Even for methods that draw k samples, there may be ways to retain the prefilled KV-cache for the image, whereas the proposed method presumably cannot reuse the image KV-cache across scales. The paper provides very little analysis of the computational overhead relative to the performance gain.

  • Overall, the experiments are limited to the Qwen2.5-VL model family. Given the many strong GUI-finetuned models that have been released recently, it is unclear whether the proposed method would yield improvements on those models as well.

Questions for authors

Please refer to the Weaknesses section. In addition, in what kinds of cases does multi-scale prediction typically flip an incorrect prediction into a correct one? Some qualitative analysis of these cases would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.