acceptodds
Withdrawn from ICLR 2027

Lazy MCFV: Observation-Aware Selection with Exact Query Pruning for Multi-view GUI Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

GUI grounding connects natural-language instructions to actionable locations in interface screenshots. Multi-view observation and magnification improve the perception of small interface elements, yet different views may produce conflicting predictions. Existing aggregation methods treat these predictions as isolated coordinates and discard the observation context from which they originate. We argue that the generating view is an essential component of a grounding prediction. Minimal Common Field-of-View (MCFV) is a training-free, observation-aware selection rule that preserves this provenance by pairing each candidate with its generating view and selecting the finest view whose observation scope contains all candidate locations. It automatically retains detailed local observations when they are sufficient and broader contextual views when ambiguity requires additional coverage. The same containment structure enables Lazy MCFV to achieve exact query pruning without changing the selector's output. We establish exact output preservation and show that the required observations are determined by the selector itself rather than heuristic confidence estimates. Experiments on ScreenSpot-Pro demonstrate improved selection from shared observations and output-preserving query reduction without additional training or verification.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
GUI grounding, training-free inference, multi-view selection, adaptive computation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_sVKtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MCFV (Minimal Common Field-of-View), a training-free selection rule for multi-view GUI grounding. Given a frozen grounding model, it first predicts a point on the full screenshot, then crops three nested windows with side ratios of 1/8, 1/4 and 1/2 around that point and queries the model on each. MCFV keeps every candidate paired with the view that produced it and selects the finest view whose window contains all candidate points, returning that view’s prediction, which automatically balances detail against context. The same containment structure yields Lazy MCFV, which queries views from coarse to fine and stops as soon as an observed point falls outside the next finer window; the authors prove that it returns exactly the same output as the full rule and that every query it issues is necessary. On ScreenSpot-Pro with GUI-Actor-7B and Qwen2.5-VL-7B, MCFV outperforms mean, medoid and token-confidence aggregation on the same observations and beats UI-Zoomer and AutoFocus with far fewer requests, while Lazy cuts model requests by about 16% with identical outputs.

Critical strengths

The most notable strength is that answer selection and query pruning follow from a single containment structure, with exact guarantees rather than heuristics. The stopping condition of Lazy MCFV is a geometric certificate: Proposition 1 proves that both the coordinate and the selected scale match the full selector, and Proposition 2 shows that any always-correct deterministic algorithm must issue every query Lazy MCFV issues. The experiments confirm this: all 1,581 outputs agree between Full and Lazy execution, whereas confidence-based and agreement-based early stopping on the same recorded predictions alter tens to hundreds of outputs. Lossless pruning of this kind is rare among inference-time acceleration methods.

Critical weaknesses

Weakness 1 Limited novelty. The pipeline is a standard native-anchored multi-scale crop scheme that is essentially the same as existing iterative zoom and multi-view methods (Iterative Narrowing, GUI-Lens, MVP, etc.); what is new is a single containment rule and the early-stopping condition it implies. Propositions 1 and 2 are nearly immediate for nested windows, and the exactness is only relative to the MCFV heuristic itself, which the paper never justifies as a principled proxy for correctness. The gain attributable to the new rule is small: 1.8 to 3.9 points over medoid, and 1.27 points over max token score in the point setting with an interval crossing zero, while most of the improvement comes from cropping itself (a single fixed 1/8 crop already lifts the box setting from 18.47% to 41.56%); Lazy pruning saves only about 16% of requests. The paper also neither compares with nor discusses trained iterative zoom methods such as GUI-Spotlight.

Weakness 2 Questionable evaluation validity. Only one benchmark and two 7B models are used. Appendix B.3 acknowledges that method development used labels from 1,016 of the 1,581 tasks and that the final evaluation reuses the same population with no held-out split, and the intervals are not adjusted for method selection, so the crop ratios and the rule itself may be tuned to this benchmark. All baselines are the authors’ own HF reproductions with mismatched image budgets and prompts; the reproduced UI-Zoomer (39.53%) and MVP (32.64%) fall below a single fixed 1/8 crop, which raises doubts about whether they were properly tuned. Finally, the rule cannot recover from a wrong coarse prediction (Fig. 5b), reverses on large targets, and it remains unknown whether it still helps with stronger grounding models.

Questions for authors

See Weaknesses

Reviewer_PDtUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Minimal Common Field-of-View (MCFV), a training-free selection rule for multi-view GUI grounding. A frozen model predicts a location on the full screenshot, which anchors three nested crops. MCFV selects the finest view containing all local predictions, while Lazy MCFV queries views coarse-to-fine and stops when containment fails. The authors prove that Lazy MCFV produces identical outputs to Full MCFV and is query-minimal under specified assumptions.

On ScreenSpot-Pro, MCFV improves over medoid selection by 1.77–3.92 points across three settings. Lazy MCFV preserves all outputs while reducing model requests by approximately 16% and serial latency by 12–17%. Additional experiments compare against existing methods and analyze stopping behavior.

Critical strengths
  1. Simple and training-free. MCFV incorporates view provenance into prediction selection without additional training, confidence thresholds, or learned verifiers.
  2. Clear theoretical guarantees. The paper establishes exact equivalence between Full and Lazy MCFV and query minimality under explicit assumptions.
  3. Careful empirical analysis. Shared-observation controls, paired bootstrap intervals, independent Full/Lazy executions, and failure-case analyses support the findings.
Critical weaknesses
  1. Limited novelty. MCFV is essentially a coarse-to-fine stopping rule, and Lazy MCFV follows naturally from it. The theoretical guarantees are relatively straightforward and apply only to this specific selector.
  2. Insufficient justification. Incorrect coarse-view predictions can override correct fine-view predictions. Fixed crops anchored to Native also prevent recovery from large initial errors. Comparisons with recentering or confidence-aware alternatives are missing.
  3. Limited evaluation and baseline concerns. Experiments cover only one benchmark and two 7B models. Development on 1,016 label-exposed evaluation tasks raises concerns about test-set tuning. The unusually weak MVP reproduction also calls baseline reliability into question.
  4. Practical efficiency and presentation issues. Latency gains are measured only under serial inference, without comparison to batched or parallel execution. Several citations in the Related Work section appear as unresolved question marks (“??” or “(?)”), and the AI Use Statement contains template placeholders.
Questions for authors
  1. Given MCFV's equivalence to coarse-to-fine stopping, what distinguishes it from existing cascaded zoom methods? How does it compare with recentered or confidence-aware alternatives under matched budgets?
  2. Could you report results on the 565 label-unexposed tasks and an additional benchmark, preferably with stronger grounding models, to establish generalization?
  3. How do Full MCFV with batched or parallel inference and Lazy MCFV compare in wall-clock latency and throughput? Does the efficiency advantage persist?
  4. How do you explain the low MVP reproduction accuracy relative to fixed-crop baselines? Could you verify the reproduced baseline results against the original papers?
Reviewer_VmFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free MCFV selection rule that uses the native prediction as an anchor to construct three nested cropping windows (1/8, 1/4, and 1/2). It selects the prediction from the finest-grained view that contains all local candidate points across the different views. Building on the same hierarchical containment structure, the paper further introduces Lazy, a coarse-to-fine query-pruning strategy that guarantees strictly unchanged outputs. On ScreenSpot-Pro, MCFV achieves an approximately 2–4 percentage-point improvement over the medoid baseline, while Lazy reduces the number of queries by approximately 16%.

Critical strengths
  1. Simple, training-free, and reproducible design with clean shared-observation controls. The paired bootstrap analysis of optionally correct candidates further strengthens the evaluation. The authors also transparently report unfavorable findings: the confidence interval for the point-based comparison with token score crosses zero, the largest-area bin shows a reversal (−7.27), and performance is tied in the Scientific domain.

  2. A unified selection-and-pruning framework with thorough validation. Full and Lazy achieve identical coordinates and scales across 1581 tasks, complemented by independently measured real execution latency. Controlled replay experiments against confidence- and consistency-based early stopping are also convincing, which highlights the fundamental distinction between a geometric certificate and heuristic early stopping.

  3. Appropriately scoped theoretical claims. The authors clearly acknowledge the limitations of their theoretical results, including counterexamples involving medoid selection and equal-sized windows, and explicitly clarify that their theoretical guarantees should not be interpreted as guarantees of actual execution latency.

Critical weaknesses

1. Incomplete citations (Lines 119–125). Several references in the Related Work section are unresolved ("(?)" and "(??)"), including those for adaptive computation time, early-exit networks, confidence-based exiting, selective prediction, and speculative decoding.

2. The claim that "the generating view is an essential component of a grounding prediction" is not isolated by any ablation. The crop windows are fully determined by the Native prediction and the image size (Sec. 3.1, Appendix B.1), so MCFV can be read as a hierarchical consistency test whose tolerance radius is the window scale. The coordinate-only controls in Table 1 (mean, median, medoid, maximum token score) ignore crop geometry, and the finest-/coarsest-view controls use view size but not the containment criterion (and are reported only for Qwen2.5-VL-7B). The gain may therefore stem from a scale prior plus a consistency check rather than from retaining provenance.

3. The complete-method comparison in Table 2 is of limited reliability. The authors concede that "the same output label does not imply identical image budgets or prompts across independently evaluated methods" (Sec. 5.3); it seems that the headline margins (+7.02 to +8.10 points over UI-Zoomer, Table 9; +4.55 points over AutoFocus) cannot be attributed to the selection rule alone. The 75.43% request reduction relative to AutoFocus (5,184 vs. 21,098) mainly reflects AutoFocus's own sampling and refinement design, not Lazy pruning, whose isolated effect is 15.91–16.75% (Table 3). The Requests/task entry for MVP is empty because its forward count (6.00 per task, Appendix C.3) uses a different unit, so no request-level efficiency comparison with MVP is supported.

4. The selection rule is bounded by the Native anchor, and neither the crop ratios nor K are ablated. Windows are centered on Native and never re-centered (Sec. 3.1), and Native accuracy is only 18.47% (Qwen box) and 27.07% (Qwen point). In the box setting, 714 of 1,581 tasks have no correct local candidate (Sec. 5.6), capping any selector at about 54.8% (867/1,581). The Conclusion acknowledges the fixed anchor, but the gap to re-centering approaches such as AutoFocus and GUI-ARP is neither explained nor quantified. Moreover, Sec. 5.5 states that its comparisons "vary view count within the fixed scale schedule, not the crop ratios themselves," so robustness to the ratios {1/8, 1/4, 1/2} and to K = 3 remains unknown.

Questions for authors

See weaknesses.

Reviewer_mhVSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes MCFV, a training-free rule for choosing among predictions from three nested crops centered on the model’s full-image prediction: it returns the prediction of the smallest crop that contains all crop predictions. Lazy MCFV queries crops from coarse to fine and stops once a prediction rules out all finer crops. The authors prove this gives the same output as querying all crops, and that its queries are necessary under their response model. On ScreenSpot-Pro with GUI-Actor-7B and Qwen2.5-VL-7B, MCFV improves over medoid selection by 1.8–3.9 points and Lazy saves ~16% of model calls.

Critical strengths
  • The core idea is simple and sensible. Treating the generating view as part of a prediction is a genuinely useful observation, and the selection rule needs no training, thresholds, or extra calls. The shared-observation controls (medoid, median, token score, finest/coarsest) isolate the selection effect cleanly.
  • The exact-pruning result is clean and verified in practice. Full and Lazy runs agree on all 1,581 tasks, and the stopping-rule replay nicely shows that confidence and agreement stopping change many outputs while Lazy changes none.
Critical weaknesses
  • Important related work is missing. MCFV’s nested, prediction-centered crops are very close to Iterative Zooming/Narrowing from the ScreenSpot-Pro paper and to Nguyen (2024). Prediction-aggregation methods such as ReGUIDE (KDE over predictions) and GUI-RC (spatial voting) are also absent, as is DiMo-GUI. None of these are discussed or compared, so it’s hard to judge how much MCFV adds beyond a coarse-to-fine cascade with a containment check.
  • The method was developed on the test benchmark. Appendix B.3 says method development used labeled observations from 1,016 of the 1,581 ScreenSpot-Pro tasks, and final results are reported on the same population. With gains over medoid of only 2–4 points, and a non-significant gap against token-score selection in the point setting, this undermines confidence in the results. Evaluating on held-out benchmarks (ScreenSpot-v2, UI-Vision, OSWorld-G) would address this.
  • The evaluation is narrow, and the comparisons are hard to trust. Only one benchmark and two 7B models are used, with no stronger recent grounders. The baselines are the authors’ own HF re-implementations under different budgets and prompts. The efficiency gain from Lazy itself is modest (~12–17% time); most of the speedup over AutoFocus comes from comparing a 3-view method against a much heavier pipeline.
  • The theory is overclaimed, and the paper looks unfinished in places. Both propositions follow almost directly from nesting, and the lower bound holds only over an adversarial product space that real model responses don’t resemble, so “information-theoretically necessary” oversells it. Section 2 has broken citations (“??”, “?”). The AI-use statement still contains unedited template placeholders, and it says AI assisted in writing the proofs.
Questions for authors
  • How does MCFV compare with Iterative Zooming/Narrowing (ScreenSpot-Pro), ReGUIDE, AFRAgent, and GUI-RC under the same backbone and request budget?
  • What are the results restricted to the 565 tasks not used during development, and on other benchmarks?
  • How often does a wrong Native prediction place all crops away from the target? Would the method still help with a stronger Native model?
  • Does MCFV still help with stronger recent grounders (e.g., Qwen3-VL, UI-TARS-1.5)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.