acceptodds
Withdrawn from ICLR 2027

SCOPE: Spatial Confidence and Offloading with Per-Element Guarantees for Device-Cloud GUI Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

GUI grounding turns a natural language instruction into a coordinate on the screen. Since cloud inference is costly, practical deployments run lightweight edge agents on the device. The edge agent closes the capability gap by offloading the tasks it would get wrong, which calls for uncertainty estimation and a risk-controlled offloading decision. Existing methods typically estimate the uncertainty from logit statistics or repeated sampling, and offload by thresholding the score. However, these methods measure hesitation implicitly without quantifying the deviation of the prediction from the target element, which decides the correctness of grounding. We instead expand the logits of the grounding coordinate digits into a click distribution over the screen, and observe that the hesitation separating correct from wrong predictions lies at the high-order digits that decide the element rather than at the low-order digits, and that wrong groundings often escape to the neighborhood of the target. Building on this, we propose SCOPE, a framework for Spatial Confidence and Offloading with Per-Element Guarantees. SCOPE first reads the separation of hesitation across coordinate scales through an entropy gate, cross-checks the prediction by re-grounding on a local view, and calibrates the combined spatial confidence into a conformal click radius in units of UI elements. The edge offloads with a per-element guarantee whenever the radius exceeds one element. Experiments demonstrate that the spatial confidence recovers on average 7.7 percentage points more of the device-cloud capability gap than the strongest baseline, and the calibrated radius offloads 58% of inputs at 90% nominal coverage.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI grounding, device-cloud collaboration, uncertainty estimation, conformal prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_FGqSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies device-cloud GUI grounding, where an on-device model offloads uncertain predictions to a cloud model. The proposed method, SCOPE, combines two scores. The scale-profile score measures the variation of top-1/top-5 log-probability gaps across coordinate digits, gated by the mean decoding entropy. The view-consistency score measures the displacement between the original prediction and a re-grounded prediction on a local crop. The combined score is calibrated with split conformal prediction into a click radius measured in target-element radii, and inputs whose radius exceeds one are offloaded. Across three benchmarks and three edge models, the method achieves a mean AUCG of 0.785, versus 0.708 for the strongest baseline, and offloads 58.4% of inputs at α = 0.1.

Critical strengths
  1. The scale-profile score reuses decoding logits without extra inference and achieves a mean AUCG of 0.720, outperforming all baselines. The full method adds 0.20× extra inference cost. Experiments cover 9 configurations and 100 random splits.

  2. Calibration normalized by element size, and useful failure analysis. Expressing the conformal radius in units of the target-element radius is meaningful. Table 12 shows that wrong predictions confirmed by re-grounding account for 37.9% of all errors but 81.0% of the errors retained on-device.

  3. The paper is well written, and easy to follow

Critical weaknesses
  1. PSS already uses coordinate-digit logits for confidence estimation. AutoFocus (arXiv:2605.02630) uses coordinate-token perplexity together with crop-based re-grounding. PACE (arXiv:2609.24277) explicitly weights per-digit entropy by place value. Neither AutoFocus nor PACE is cited or compared. Moreover, the steepness term gg is a standard deviation across digit positions and is therefore invariant to permutations of those positions. It cannot distinguish uncertainty in higher-order digits from uncertainty in lower-order digits, which is the paper's own motivation. Comparisons with simple alternatives, such as coordinate-only entropy or place-weighted entropy, are missing.

  2. The guarantee does not match click correctness, which tilts the SafeGround comparison in SCOPE's favor. Eq. (3) bounds P(δ=0∧dist(p^e,B)>w)P(\delta=0 \wedge \mathrm{dist}(\hat p_e, B) > w), where ww is half the box diagonal. Clicks outside the target box but within ww of it are therefore not counted as violations, even though correctness requires clicking inside the box. The guarantee is also marginal over all inputs and does not bound the error rate among retained predictions. SafeGround, by contrast, controls the false discovery rate (FDR) among accepted predictions, which is a stricter guarantee. Applying the proposed rule to SafeGround's scores reduces its offloading rate from 94.2% to 74.3% (Table 3). Meanwhile, SafeGround's native rule retains predictions with higher accuracy (0.911 vs. 0.891 for SCOPE). Much of the reported 58% versus 94% gap therefore stems from the different guarantees rather than from better uncertainty estimation.

  3. The deployment results contradict the device-cloud motivation. Table 4 shows that cloud-only inference is more accurate than SCOPE (0.782 vs. 0.773), uses less computation (27.9 vs. 46.5 edge + 18.3 cloud TFLOPs), and has lower latency (3.99 vs. 5.98 s). The scale-profile-only fallback suggested in §4.4 does not fix this: at its 74.3% offloading rate, we estimate its latency at roughly 2.17 + 0.743 × 3.99 ≈ 5.1 s, which is still slower than cloud-only. On ScreenSpot-Pro, 97.3–100% of inputs are offloaded (Table 7). The chosen cloud model activates only about 3B parameters per token, and no monetary cost comparison is provided.

Questions for authors
  1. What proportion of incorrect predictions satisfies 0<di≤wi0 < d_i \le w_i and therefore does not violate the constraint in Eq. (3)? What is this proportion among the retained predictions?

  2. Can the authors compare with SafeGround under the same type of guarantee, either by applying FDR-controlled calibration to their scores or by comparing retained accuracy at matched offloading rates?

  3. Can the authors compare against PACE, coordinate-token-only entropy, and a higher-order-digit-only score? How does the scale-profile score behave on samples whose higher-order digits are uncertain but whose lower-order digits are confident?

  4. Why was Qwen3.5-35B-A3B chosen as the cloud model? Would the method still reduce cost with a substantially stronger and more expensive cloud model, when monetary cost is measured?

Reviewer_WwG4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SCOPE, a framework for uncertainty-aware GUI grounding that combines spatial confidence estimation with conformal calibration to guide device-cloud offloading, aiming to reduce cloud usage while maintaining reliable grounding.

Critical strengths

The paper addresses uncertainty estimation and risk-controlled offloading for GUI grounding, which is a practically relevant problem. And the presentation of this paper is fine.

Critical weaknesses
  1. Lmited novelty and substantial overlap with SafeGround, overall framework of spatial uncertainty estimation, conformal calibration, and selective GUI grounding very closely resembles SafeGround. The authors should more clearly distinguish their fundamental contributions beyond replacing the uncertainty estimator and calibration rule.

  2. Potentially misleading theoretical guarantees. The proposed per-element guarantee controls the distance between the prediction and target bounding box, rather than whether the predicted click actually falls inside the correct element. Moreover, the guarantee is marginal rather than conditional on individual UI elements. It is therefore unclear whether the claimed guarantee adequately reflects grounding correctness.

  3. SCOPE and SafeGround control different risk quantities. Comparing their offloading rates at the same nominal risk level may therefore be misleading without evaluating both approaches under a common correctness-based risk constraint.

  4. In Section 3.2.2 and Algorithm 2, the normalization statistics (median and MAD) are estimated from the same calibration set used to compute the conformal quantile. This data-dependent score construction may violate the exchangeability requirements of standard split conformal prediction. The authors should clarify whether the finite-sample coverage guarantee remains valid without an independent fitting set.

  5. In Appendix A.1 (Equations 17–22), the authors multiply probabilities from different coordinate-digit positions to construct candidate coordinates. However, these probabilities are obtained from autoregressive decoding and are conditioned on the original token prefix. Combining alternative digits independently does not necessarily represent a valid joint distribution over coordinates. The authors should justify this approximation and evaluate its impact on uncertainty estimation.

Questions for authors
  1. What is the fundamental conceptual difference between SCOPE and SafeGround, beyond the proposed uncertainty estimator and conformal calibration formulation?

  2. Since a prediction can fall outside the target bounding box while satisfying the proposed distance-based coverage guarantee, how does the method guarantee the correctness of clicks retained on the device?

  3. Could the authors compare SCOPE and SafeGround under the same risk definition, particularly the error rate among retained predictions, to establish whether SCOPE offers a genuine efficiency advantage at matched reliability?

Reviewer_ehRsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCOPE, a framework for uncertainty estimation and device-cloud offloading in GUI grounding.

Critical strengths

The paper presents an intuitive approach to uncertainty estimation. It uses the probabilities of different coordinate digits to estimate spatial uncertainty and obtains an additional consistency signal through local re-prediction. The experimental results also show that the proposed approach is effective.

Critical weaknesses
  1. The paper defines its risk-control objective based on whether the distance between the predicted point and the target bounding box is within one element radius. However, in GUI grounding, a prediction is considered correct only when the predicted point falls inside the target bounding box. These two conditions are not equivalent. A prediction outside the target box may still satisfy the risk-control condition if it is sufficiently close to the box. Therefore, the theoretical guarantee does not directly ensure correct GUI clicks.

  2. The paper constructs a candidate click distribution by multiplying the probabilities of different coordinate digits. However, in an autoregressive model, the probability of each digit depends on the preceding digits. When earlier digits are changed to form alternative coordinates, the probabilities of later digits may also change. Therefore, the constructed distribution may not accurately represent the probabilities of these alternative coordinates. Moreover, Figure 6 shows that the number of candidates has little effect on performance. It would be useful to compare against a simpler baseline that directly crops around the initial predicted click, using the same crop size, and performs a second prediction. This would help clarify whether the improvement comes from the additional spatial information provided by the candidate distribution or simply from local cropping and re-prediction.

  3. Although SCOPE reduces cloud inference and data upload, it requires an additional local inference step, and the edge and cloud inference stages run sequentially. As shown in Table 4, SCOPE has a highe latency than Cloud-only.

Questions for authors

See Weaknesses

Reviewer_7si6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SCOPE, a training-free approach for uncertainty estimation and offloading in GUI grounding. It combines coordinate-token confidence with consistency between predictions on the original and cropped screenshots. The resulting confidence is calibrated using conformal prediction to determine whether to retain the edge prediction or offload to a cloud model. The method is evaluated across three GUI benchmarks and three edge models, showing improvements over existing uncertainty-estimation baselines.

Critical strengths

Using coordinate-level uncertainty, particularly the distinction between high-order and low-order digits, is an interesting direction compared to conventional token-level confidence.

The approach is training-free and relatively straightforward to integrate with existing GUI grounding models.

The evaluation is fairly comprehensive, covering nine model-benchmark configurations. The improvement in mean AUCG from 0.708 to 0.785 is meaningful. The ablations and deployment analysis are also useful.

Critical weaknesses

Limited novelty. PSS already explores coordinate-logit uncertainty and cropping, while SafeGround considers spatial uncertainty and calibrated offloading. The paper does not clearly establish the novelty beyond combining these ideas.

Scale-profile formulation. The motivation emphasizes uncertainty in high-order versus low-order coordinate digits, but the proposed standard deviation is invariant to digit ordering. It is unclear whether this statistic captures the intended spatial uncertainty.

Interpretation of the guarantee. The conformal formulation bounds marginal errors exceeding one element radius, rather than directly controlling incorrect clicks or the error rate among predictions retained on the device. This is weaker than the practical reliability objective.

Contribution of the second inference. The scale-profile score achieves 0.720 AUCG, while adding cropped-view consistency increases it to 0.785. It is unclear how much of this gain comes from the proposed confidence formulation versus simply performing another grounding pass.

Practical efficiency. SCOPE reduces cloud usage but increases end-to-end latency (5.98s versus 3.99s for cloud-only inference). The accuracy-latency tradeoff is not entirely convincing.

Confidently incorrect predictions. The method struggles when the original and cropped predictions agree on the same wrong element. Such cases account for 81% of incorrect predictions retained on the device, limiting the reliability of the approach.

Questions for authors

How does SCOPE differ substantially from PSS and SafeGround? Can the authors compare against existing confidence methods with the same cropped-view inference and calibration?

Why use the standard deviation of coordinate-digit confidence gaps when it does not preserve digit ordering? Would a position-aware statistic better capture the proposed motivation?

Why calibrate errors exceeding one element radius rather than directly controlling incorrect clicks among retained predictions?

Can the authors better isolate the gains from the proposed uncertainty score versus the additional cropped-view inference?

Given the higher latency than cloud-only inference, what deployment scenarios offer a meaningful accuracy-cost advantage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.