acceptodds
Under review as a conference paper at ICLR 2027

VeriAct: A Certificate Veto Costs Coverage on Unverifiable Actions, Not on Failing Ones

Abstract

A computer-use agent that must act at every step inherits the full error rate of its model. We study a training-free gate between a frozen GUI agent's policy and the environment that decides, for each proposed action, whether to ACT, ABSTAIN, or ASK a human. The gate fuses three uncertainty signals (self-consistency over equivalence clusters of the sampled actions, a frozen pre-operative critic, and model confidence), applies a machine-checkable static executability certificate over the accessibility tree or DOM as a veto, and turns the fused score into an ACT threshold whose executed-step error is bounded by simultaneous Bentkus bounds over a fixed threshold family, fitted on independent episode anchors. On AndroidControl and Mind2Web's three splits with two frozen 7B backbones the gate attains an 8.9% mean AURC reduction over the best of its own component signals used alone, with no external calibrated gate compared, and the pooled certificate keeps executed risk at or below the target at every non-vacuous setting. The main finding concerns the veto. 72.9% / 66.1% of what it withholds on AndroidControl and on Mind2Web carries no applicable check at all rather than a check that fails. What the veto buys is that no proposal violating the static checker's preconditions executes, and withholding only that checker-failing slice keeps the certificate and recovers certified coverage on both benchmarks, most of it on AndroidControl. The certified level does not transfer across splits although the ordering of policies does, and all claims are offline and step-level, with no live execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
computer-use agents, GUI agents, conformal risk control, abstention, action verification, certified coverage

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_h8y7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors present VeriAct, a framework that gates the interactions between a GUI agent and the environment.

Critical strengths

The problem setting itself is something that I think is valuable. I agree that having a risk-coverage based method to gate actions is important for GUI agents in deployment.

Critical weaknesses

The certificate only seems to be useful at 30% error, which is nowhere near practical for the kinds of failures that the paper itself describes. In addition, the threshold acceptance sets seem like they would be better off not being treated as independent, as the specific certificate style used seems to be the bottleneck there in terms of requiring a lot of calibration data. Finally, the different splits seem to vary the risk percentage notably, meaning the distribution itself used for the certificate seems to be artificial.

In addition, the paper overall presents a lot of negative results (which I think can be good in moderation!) and a lot of hedging language that inevitably cuts away at much of the empirical strength of the paper. For example, results seem to support Policy B, not Policy A, the deployment configuration proposed by the authors.

Questions for authors
  1. Your own Table 3 strongly favors withholding checker failures but releasing "no-check" actions. Why isn't Policy B the proposed method?
  2. Can you compare against other predictors rather than ablations of your own components?
  3. Why would a=0.30 be acceptable for the types of irreversible actions that motivate something like this work?
2 comments
AuthorsOpenReview ↗

Response to Reviewer h8y7

Thank you for the review. Numbers marked "post-review" come from a re-analysis of the stored records after we read the reviews. The revision prints them in Appendix A.3 (Tables 7 to 12). They are exploratory and change no registered result.

Why is Policy B not the proposed method? (Question 1) Your reading of Table 3 is right. At alpha=0.3, Policy B moves coverage from 38.3% to 55.3% on OS-Atlas and from 67.4% to 69.6% on UI-TARS, at executed risk 28.4% and 27.8% (Table 7). It still blocks every checker failure. Post-review, we found two limits. Its lower threshold admits certified proposals that err more often (197 at 65.0% on OS-Atlas, where its risk interval contains alpha). Calibrated on AndroidControl alone, Policy A certifies UI-TARS at alpha=0.1 and Policy B does not. The revised conclusion states that Policy B raises coverage at the three non-vacuous settings of the evaluated mixture, and that this does not establish that it is better under per-slice calibration or in live execution. Policy A stays in the paper as the configuration under study; we did not show that its requirement of a passing check has practical value.

Other predictors (Question 2). Post-review, we compared the gate with an agreement entropy (a surrogate for semantic entropy over our action equivalence) and mean sample confidence, both computed from the same samples (Table 8). The gate's AURC is lower than that of the agreement entropy by 0.0325 [0.0175, 0.0475] on AndroidControl and by 0.0346 [0.0147, 0.0538] on Mind2Web. Over 8 legs, its advantage excludes zero on 3 against the agreement entropy and on 8 against mean sample confidence, and it is worse on none. External calibrated gates are still not compared.

Alpha=0.30 for irreversible actions (Question 3). It is not acceptable for those actions, and the revised limitations say so. The lowest certified target is alpha=0.2 on OS-Atlas, at 8.7% coverage with executed step risk 3.8%. With the simultaneous bound on the pooled mixture and 952 anchors, no lower target is certified. Post-review, we estimated the anchors needed at the observed rate (Appendix A.3): OS-Atlas needs 1,950 at alpha=0.1 and 8,659 at alpha=0.05. Two other analyses reach alpha=0.1 but give no pooled guarantee: the exploratory fixed-sequence rule below (8.7% coverage on OS-Atlas) and calibration on AndroidControl alone, which certifies UI-TARS on AndroidControl only (43.7% coverage, executed risk 8.6%).

Treating the threshold sets as independent. The simultaneous bound does not assume independence: the union bound needs none, and the revised method section says so. It does ignore nesting, and your suggestion helps. Post-review, we tried a fixed-sequence rule that uses the nesting (Table 11). On the published split, coverage goes from 0.0% to 8.7% at OS-Atlas alpha=0.1, from 8.7% to 21.4% at alpha=0.2, from 38.3% to 41.7% at alpha=0.3, and from 67.4% to 74.5% at UI-TARS alpha=0.3. Its starting threshold depends on the calibration scores, and we have no proof that it keeps the error rate at delta, so the revision reports it as exploratory. On 200 resplits, under the reading that we registered (resplits with a non-empty certificate), anchor risk exceeds alpha in at most 10% at 5 of 6 settings and in 20.0% (3 of 15) at OS-Atlas alpha=0.05, where the published split is vacuous. Divided by all resplits (a post hoc reading), the largest rate is 8.0%. Step-weighted risk exceeds alpha in up to 54.2% of the non-empty resplits. The certified method stays the simultaneous bound.

Variation across splits. The certificate is marginal over the pooled mixture, and the paper states that the certified level does not transfer across splits. Post-review, per-slice calibration under Policy A is vacuous in 27 of 32 cells (Table 10), so these data cannot support a per-split guarantee.

Negative results and qualifiers. The revision narrows the stated contribution: the abstract and introduction now present the gate as the configuration under study in an offline diagnosis of what it certifies and what its veto costs, with ASK as a routing rule whose value we do not measure. The conclusion now states what Table 3 shows for Policy B, together with its limits. The qualifiers that remain state the scope of the claims (offline, step-level, marginal over the pooled mixture).

Thank you for the response. While the revised framing fits the paper better, it does not resolve my concerns about the paper's contribution. I therefore firmly maintain my original rating of reject.

Reviewer_RVDkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free gate that selects ACT, ABSTAIN, or ASK for actions proposed by a frozen GUI agent. It combines self-consistency, critic scores, and model confidence with statistical risk calibration and a static-checker veto. The evaluation examines risk–coverage trade-offs and the coverage cost of this veto on AndroidControl and Mind2Web. On a lighter note, the manuscript is unusually eager to present its results: Figure 1 appears before even the title.

Critical strengths
  • Although it is not novel, deciding when a GUI agent should refrain from acting is an important problem.

  • Distinguishing checker-failing actions from actions for which no check applies is a useful analysis.

Critical weaknesses
  1. The presentation lacks basic structural clarity. Figure 1 precedes even the title, presenting certified operating points and a veto-policy comparison before the reader has encountered the problem, method, or meaning of the certificates. This is a presentation concern rather than a technical objection, but it makes the manuscript feel insufficiently organized and polished.

  2. The central hard-veto design is not adequately justified. Passing the static checker establishes only certain preconditions on the recorded accessibility structure; it does not establish semantic correctness or successful execution in the live environment (§3.3). Conversely, the absence of an applicable check does not establish that an action should be rejected. Table 3 shows that allowing these unverifiable actions and recalibrating the gate preserves the statistical risk guarantee while increasing coverage. The authors describe the deployed veto as a design commitment, but do not establish the practical value of the additional property it enforces. A central component therefore remains a costly restriction without a demonstrated benefit commensurate with that cost.

  3. The additional value of the three-way decision is not established. ASK is determined by a fixed score band and an irreversibility taxonomy, but the paper does not establish why this rule identifies actions that particularly warrant human intervention. On Mind2Web, 95.1% of records have unknown irreversibility and none are classified as irreversible, leaving the taxonomy largely untested. At the same ASK budget, selecting the lowest-scoring actions captures errors more effectively (Table 6). This does not establish that the proposed routing is worse at preventing harm, but harm prevention and intervention utility are precisely what the evaluation does not measure. Consequently, the contribution of ASK beyond a binary selective gate remains unsupported.

  4. The connection between the certified quantity and the quantity of practical interest is incomplete. The guarantee concerns accepted-anchor error under a pooled mixture, with one anchor selected per episode. This differs from step-weighted executed-action error and does not control risk within individual environments. Avoiding within-episode dependence is reasonable, but it does not establish that the resulting episode-weighted guarantee controls these other quantities. The authors acknowledge this scope; the missing contribution is evidence that the narrower guarantee is sufficiently useful. Its applicability is also limited: five of eight settings in Table 2 are vacuous, as are all 24 single-split calibration settings in §4.3.

Overall, the experimental evaluation is not sufficiently rigorous to establish the full system’s added value, particularly given the absence of external calibrated-gate comparisons and direct evaluation of execution or intervention outcomes.

Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer RVDk

Thank you for the review. Numbers marked "post-review" come from a re-analysis of the stored records after we read the reviews. The revision prints them in Appendix A.3 (Tables 7 to 12). They are exploratory and change no registered result.

Figure 1. Fixed. In the revision it prints after the title and the start of the introduction.

The hard veto. Your objection stands. Table 3 recalibrates both policies under the same target, and post-review we looked at what Policy B releases (Table 7). It executes 226, 262 and 58 unverifiable proposals at the three non-vacuous settings, and they err at 6.2%, 8.4% and 3.4%. Its lower threshold also admits certified proposals that err more often: 342 at 25.1% (OS-Atlas, alpha=0.2) and 197 at 65.0% (alpha=0.3). At OS-Atlas alpha=0.2, Policy B moves coverage from 8.7% to 29.7% and executed risk from 3.8% to 13.6%, still below the target. At alpha=0.3 its interval, [26.0, 31.1]%, contains the target. Policy B is also not better everywhere: calibrated on AndroidControl alone, Policy A certifies UI-TARS at alpha=0.1 and Policy B is vacuous. Policy A adds one property: every executed proposal passed an applicable check. We did not show that this property improves outcomes. The revised conclusion states that Policy B raises coverage at the three non-vacuous settings of the evaluated mixture, under the same marginal anchor-risk target, and that this does not establish that it is better under per-slice calibration or in live execution.

ASK. We agree. The paper does not establish the added value of the three-way decision, and it measures neither harm prevention nor intervention utility. The revision now presents ASK as a routing rule whose value we do not measure.

Certified quantity and practical quantity. The revision names the certified quantity as executed-anchor error in the abstract, introduction and conclusion, and it labels step-weighted numbers as step-weighted (for example, the seed re-draw read-out in the appendix). Post-review, we also computed step-weighted executed risk on the pooled test records (Table 12): 3.8% [1.6, 6.5] at OS-Atlas alpha=0.2, 26.6% [23.7, 29.4] at OS-Atlas alpha=0.3, and 28.5% [26.3, 30.8] at UI-TARS alpha=0.3. The anchor risks are 3.0%, 24.6% and 25.9%, and the bounds are 13.9%, 29.1% and 29.7%. So the point estimates are also below the target on this mixture, and one interval (UI-TARS) extends above it. This is a measurement and not a guarantee, and it does not control risk within one environment.

Vacuity. Five of eight settings are vacuous (Appendix A.3). We estimated the anchors that the bound needs if the observed rate stays fixed, over the thresholds that accept at least 5% of the anchors. OS-Atlas needs 1,950 anchors at alpha=0.1 and 8,659 at alpha=0.05, and UI-TARS needs 2,504 at alpha=0.2, against 952 stored. For UI-TARS at alpha=0.1 and alpha=0.05 this estimate gives no count, because the smallest observed rate over those thresholds is 14.1%. These are plug-in estimates, not guarantees.

External comparison. Post-review, we added two uncertainty scores from outside our components, computed from the same samples: an agreement entropy and mean sample confidence (Table 8). The agreement entropy equals the discrete semantic entropy of the action clusters when our action equivalence is transitive; it is not transitive on 387 of 5,404 steps, so it is a surrogate. Against it, the gate's AURC difference is -0.0325 [-0.0475, -0.0175] on AndroidControl and -0.0346 [-0.0538, -0.0147] on Mind2Web. Over 8 legs, the gate is better with an interval that excludes zero on 3 against the agreement entropy and on 8 against mean sample confidence, and it is worse on none. These are not external calibrated gates, and that comparison remains open. We also have no direct evaluation of execution or intervention outcomes.

Reviewer_SB3LOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work introduces VeriAct, a training-free selective execution gate designed to mitigate risk in computer-use GUI agents operating on mobile and web environments. Operating as a post-policy, pre-action layer, VeriAct decides whether an agent’s candidate step should be directly executed (ACT), rejected (ABSTAIN), or escalated to a human operator (ASK). The gate fuses four training-free evidence sources: a deterministic static executability certificate over the DOM/accessibility tree (SstaticS_{\text{static}}), candidate self-consistency (SscS_{\text{sc}}), a frozen pre-operative critic (ScriticS_{\text{critic}}), and token confidence (SconfS_{\text{conf}}). To guarantee step-level execution risk bounds, it applies finite-sample PAC calibration using simultaneous Bentkus upper bounds over independent episode anchors. Through offline step-level evaluations on AndroidControl and Mind2Web, the authors report AURC improvements and analyze the coverage cost associated with static executability vetoes.

Critical strengths
  1. Fusing distribution-free PAC calibration (via Bentkus bounds) with a three-way decision taxonomy (ACT/ABSTAIN/ASK) offers a mathematically principled baseline for controlling per-step error rates in computer-use agents without retraining backbone policies.
  2. The empirical analysis provides valuable insights into static executability certificates by explicitly separating checker-failing proposals from unverifiable ones. The finding that 66.1%–72.9% of withheld proposals stem from checker inapplicability rather than verified errors is a noteworthy contribution.
  3. The work demonstrates reporting discipline, including macro-generated table outputs, fixed seed replay audits, paired episode bootstrap confidence intervals, and multi seed run-to-run drift analyses.
Critical weaknesses
  1. The evaluation is conducted on static, replayed benchmark steps (AndroidControl and Mind2Web) without any live environment execution. Offline step accuracy fails to capture long-horizon trajectory dynamics, state drift, cascading error propagation, or interactive recovery.
  2. The statistical guarantees hold only marginally on the pooled mixture of calibration anchors. On individual benchmark splits or conditional slices, risk bounds are frequently violated (exceeding target risk α\alpha by up to +40.5 pp on Mind2Web). Moreover, single-split calibration is vacuous across all settings due to insufficient anchor mass.
  3. The irreversibility-aware ASK mechanism relies on a simplistic rule taxonomy (send, delete, pay) that suffers from severe dataset coverage imbalance (95.1% unknown on Mind2Web). At the gate's default operating threshold, a simple score-based rule identifies action errors better than the proposed ASK band.
  4. The static checker acts as a blunt veto that penalizes valid coordinate-free or semantic actions (such as navigating back or opening an app). This creates an aggressive coverage bottleneck where benign actions are withheld simply because no static DOM check exists.
  5. Combining heterogeneous scores (logits, discrete consistency counts, and critic outputs) via naive equal-weight averaging lacks theoretical justification. Disparate component scales can destabilize ranking quality when underlying signals are noisy or miscalibrated.
Questions for authors
  1. How does VeriAct perform in a closed-loop, live execution environment where agent actions dynamically alter the underlying GUI state and accessibility tree?
  2. Can the PAC calibration framework be adapted to provide conditional guarantees for specific task domains or web domains without requiring prohibitively large calibration datasets?
  3. How can the static executability checker be extended to evaluate coordinate-free or app-level semantic actions to prevent withholding valid, unverifiable proposals?
  4. Why was equal-weight averaging selected over learned or rank-calibrated fusion, given the varying output scales of logprobs, critic scores, and sampling clusters?
  5. How do human latency, cognitive load, and potential human error rates impact the practical utility and total task duration when operating the ASK intervention path?
2 comments
AuthorsOpenReview ↗

Response to Reviewer SB3L

Thank you for the review. The answers follow your questions. Numbers marked "post-review" come from a re-analysis of the stored records after we read the reviews. The revision prints them in Appendix A.3 (Tables 7 to 12). They are exploratory and change no registered result.

Live execution (Question 1). We have no closed-loop result. All claims are offline and step-level, so the paper does not measure state drift, cascading errors or recovery. The abstract says this, and we do not extend the claim.

Conditional guarantees (Question 2). Not at these calibration sizes. Post-review, we calibrated each slice on its own anchors (Table 10). Under Policy A, a non-empty certificate exists in 5 of 32 (slice, alpha, backbone) cells, all on AndroidControl, and none of the 5 exceeds alpha at the point estimate. Every Mind2Web slice is vacuous when it is calibrated alone. Your reading of the pooled gate is correct: applied to a slice, it executes steps in 11 cells and exceeds alpha in 7 of them, all on Mind2Web (one of the 7 has a single executed step). We also estimated how many anchors the pooled bound needs if the observed error rate stays fixed, over the thresholds that accept at least 5% of the anchors: 340 at alpha=0.3, 646 at alpha=0.2 and 1,950 at alpha=0.1 on OS-Atlas, against 952 stored (Appendix A.3). This is a plug-in estimate, not a guarantee. The stored records carry no task label and no web-domain label, so we did not compute finer slices. The revision also corrects our wording: the abstract, introduction and conclusion now say that the certificate bounds executed-anchor error, as the method section states, not executed-step error.

Extending the checker (Question 3). We did not extend it to coordinate-free or app-level actions. We measured the alternative, which is to release the proposals that have no applicable check (Policy B, Table 7). Post-review, the unverifiable proposals that Policy B executes err at 6.2%, 8.4% and 3.4% at the three non-vacuous settings. Policy B also executes certified proposals that its lower threshold admits, and these err more often: 25.1% of 342 at OS-Atlas alpha=0.2 and 65.0% of 197 at alpha=0.3. Its pooled executed risk stays below alpha at all three settings, but at OS-Atlas alpha=0.3 its 95% interval, [26.0, 31.1]%, contains alpha. Calibrated on AndroidControl alone, Policy A certifies UI-TARS at alpha=0.1 and Policy B does not. The revised conclusion states that Policy B raises coverage at the three non-vacuous settings of the evaluated mixture, under the same marginal anchor-risk target, and that this does not establish that it is better under per-slice calibration or in live execution.

Equal weights (Question 4). We chose equal weights because they need no training. Post-review, we compared rank fusion and a learned fusion (L2-regularized logistic regression, fitted on the calibration records) on the same records (Table 9). Over the 8 legs, rank fusion is better on 1, worse on 1 and not distinguishable on 6. Learned fusion is better on 0, worse on 2 and not distinguishable on 6. Mean AURC over the AndroidControl legs is 0.1373 for equal weights, 0.1451 for rank and 0.1392 for learned. On Mind2Web it is 0.5654, 0.5666 and 0.5643, and both intervals against equal weights include zero. So these records give no evidence that a calibrated fusion ranks better.

Human cost of ASK (Question 5). We did not measure it, because no person was in the loop. The paper claims no net benefit from ASK. Your points on the taxonomy's coverage (95.1% unknown on Mind2Web) and on the score-based rule are correct, and the paper reports both.

Thank you to the authors for the response and the post-review analyses.

However, rather than resolving the fundamental technical and empirical weaknesses raised in my review, the rebuttal primarily addresses them by narrowing the paper's claims and adding disclaimers. Specifically:

  1. Confining claims to offline replay does not resolve the lack of closed-loop validation under state drift and cascading errors.

  2. The post-review results further confirm that per-slice guarantees are mostly vacuous (27/32), the ASK mechanism remains completely unvalidated, and the hard veto's practical utility over releasing unverifiable actions is still unsubstantiated.

Because the core soundness and evaluation issues remain unaddressed, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.