- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies instability in GUI grounding models caused by one-pixel changes in screenshot dimensions. It identifies dynamic-resolution preprocessing transitions as a source of UI-element switching, validates the phenomenon across multiple models and benchmarks, proposes a black-box probe for inferring hidden preprocessing settings from click behavior, uses shift consistency for error detection, and introduces Grid-Phase Canonicalization (GridPC) as a lightweight mitigation.
Critical strengths
This is a careful and well-motivated study of a concrete failure mode in computer-use agents. The paper shows why average accuracy can hide instability and strengthens the diagnosis with transition-pair controls, cross-model tests, and controlled interventions. It also provides practical follow-ups: a black-box preprocessing probe, a useful consistency signal, and a lightweight mitigation that does not require training.
Critical weaknesses
The mitigation and reliability claims need stronger downstream validation. GridPC is mainly evaluated in controlled static settings, whereas real agents face browser scaling, scrolling, OS rendering differences, dynamic content, overlays, and multi-step feedback; evidence on interactive task success is missing. The switching analysis also depends on UI-element parsing and assignment rules, so more detailed robustness checks are needed to rule out parser errors or boundary choices. Finally, shift consistency requires extra forward passes and should be compared with multi-view aggregation or sampling-based uncertainty under matched compute.
Questions for authors
- How sensitive are the main switching conclusions to the UI parser and element-assignment rules?
- Does GridPC improve downstream task success in an interactive benchmark such as OSWorld?
- Under equal compute, how does shift consistency compare with multi-view or sampling-based uncertainty baselines?
7 comments
Response to Q3: shift consistency under a matched inference budget
Q3/W: Under equal compute, how does shift consistency compare with multi-view or sampling-based uncertainty baselines?
Thank you for suggesting this comparison. During the rebuttal period, we compared shift consistency with sampling-based consistency on Qwen3-VL-8B and UI-Venus-1.5-8B. Each model was evaluated on the paper's 2,010 targets: 600 from ScreenSpot-Pro, 900 from UI-Vision, and 510 from OSWorld-G.
Both methods use four forward passes and share one greedy reference prediction. Shift consistency uses three additional deterministic shift probes. Sampling-based consistency uses three independent samples of the same image at temperature 1. We report AUROC for detecting incorrect reference predictions, where higher is better. We also combine each consistency signal with the same token confidence score by averaging their risk ranks, without fitting parameters or additional forward passes.
| Error detection signal | Qwen3-VL-8B AUROC | UI-Venus-1.5-8B AUROC |
|---|---|---|
| Sampling consistency | 0.790 | 0.733 |
| Shift consistency | 0.803 | 0.752 |
| Sampling consistency + token confidence | 0.8562 | 0.7973 |
| Shift consistency + token confidence | 0.8626 | 0.8081 |
With the same number of forward passes, shift consistency achieves higher AUROC on both models, both as a standalone signal and when combined with token confidence. We will include these results and the comparison protocol in the revised paper.
Response and maintained rating
Thank you for the comprehensive response and additional analyses. The alternative element-assignment rules and human judgments strengthen the core finding, and the matched-budget uncertainty comparison directly addresses one of my questions. The MiniWoB experiments also provide useful evidence that preprocessing-induced grounding changes can propagate to task outcomes.
My remaining concern is that the downstream mitigation claim is still not fully established. The new multi-step experiment demonstrates that preprocessing transitions affect final outcomes, but it does not directly show that GridPC improves end-to-end task success in a realistic interactive benchmark such as OSWorld. Its robustness beyond controlled padding, shifts, and selected rendering changes therefore remains uncertain. I will maintain my original rating of 2 (Weak Rejection).
Response to Q1–Q2: robustness to element assignment and downstream relevance
We appreciate your constructive feedback and positive assessment of the study’s motivation and practical contributions. We hope the explanations below answer your questions. Please let us know if any points need further clarification.
Thank you for this question. The excess switching persists under human judgments and alternative element-assignment rules. Table 6 reports a blinded human evaluation of Qwen3-VL-8B on 300 screenshots, with 100 each from ScreenSpot-Pro, UI-Vision, and OSWorld-G. Each screenshot contributed one transition pair and two adjacent controls, giving 900 prediction pairs. One expert judged whether the clicks selected different UI elements without seeing the padding amounts, grid dimensions, or pair type. Excess switching is the transition switching rate minus the mean rate of the two adjacent controls. Human judgments yield 11.33 percentage points of excess switching (95% CI [7.50, 15.33]), compared with 11.17 points [7.33, 15.17] under automatic assignment.
During the rebuttal period, we reanalyzed the same human annotations using two alternative definitions of switching. Counting switches only between elements with different functions gives 21.33% switching at transitions and 10.00% for adjacent controls, an excess of 11.33 percentage points. Also counting changes between elements and background gives an excess of 14.67 percentage points. The transition effect persists under both definitions. We also compared three automatic assignment rules on the same 900 pairs, keeping the OmniParser detections and model predictions fixed:
The first two rules leave clicks outside all detected regions unassigned. All pairs remain in the denominator, and pairs with unassigned clicks do not count as switches. The table uses 10,000 bootstrap resamples of screenshots within each benchmark, with the same draws across rules.
All three rules yield positive excess switching, with 95% CIs above zero. The transition effect therefore persists under alternative assignment rules. We will add these robustness checks to the revised paper.
Thank you for this valuable suggestion. During the rebuttal period, we added two experiments on MiniWoB to explore the potential value of stabilizing preprocessing for interactive tasks. Specifically, we examined how preprocessing affects multi-step task outcomes and how GridPC behaves at different rendering scales.
For multi-step outcomes, we tested Qwen3-VL-8B on eight MiniWoB task templates under two settings: an intervention at one random intermediate click, and repeated interventions at preselected random steps. Each setting used 696 task instances, with a maximum of 30 decisions per instance. We varied screenshot padding during grounding and executed the predicted clicks in the original browser, with planning and error recovery enabled. Compared with adjacent one-pixel controls that preserve the preprocessing configuration, transitions increased final-outcome switching by 6.03 percentage points (95% CI [4.24, 7.90]) for a single intervention and 4.38 points [2.80, 6.03] for repeated interventions. Final-outcome switching counts pairs with different success or failure outcomes. The intervals use bootstrap resampling stratified by task. Identical-input repeats showed zero switching. These results show that grounding changes caused by preprocessing can propagate to final task outcomes.
For rendering scale, we evaluated Qwen3-VL-8B and UI-Venus-1.5-8B on 64 MiniWoB scenes (173 targets) rendered in Chromium at DPR 1.0 and 1.25 with unchanged DOM/CSS layout. Within each DPR, GridPC maps variants differing by one pixel of padding to identical processed inputs and yields zero switching. This demonstrates its effectiveness against one-pixel padding at both rendering scales.
On the other hand, we would like to clarify the scope of our work. Our central contribution is to identify and causally characterize preprocessing as a structural source of GUI grounding instability. GridPC serves as a simple mitigation guided by these findings, and its evaluation focuses on grounding stability. Evaluating whether GridPC improves end-to-end task success on benchmarks such as OSWorld is a valuable direction for future work. We will include this discussion and these experiments in the revised paper.