- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors present VeriAct, a framework that gates the interactions between a GUI agent and the environment.
Critical strengths
The problem setting itself is something that I think is valuable. I agree that having a risk-coverage based method to gate actions is important for GUI agents in deployment.
Critical weaknesses
The certificate only seems to be useful at 30% error, which is nowhere near practical for the kinds of failures that the paper itself describes. In addition, the threshold acceptance sets seem like they would be better off not being treated as independent, as the specific certificate style used seems to be the bottleneck there in terms of requiring a lot of calibration data. Finally, the different splits seem to vary the risk percentage notably, meaning the distribution itself used for the certificate seems to be artificial.
In addition, the paper overall presents a lot of negative results (which I think can be good in moderation!) and a lot of hedging language that inevitably cuts away at much of the empirical strength of the paper. For example, results seem to support Policy B, not Policy A, the deployment configuration proposed by the authors.
Questions for authors
- Your own Table 3 strongly favors withholding checker failures but releasing "no-check" actions. Why isn't Policy B the proposed method?
- Can you compare against other predictors rather than ablations of your own components?
- Why would a=0.30 be acceptable for the types of irreversible actions that motivate something like this work?
2 comments
Thank you for the response. While the revised framing fits the paper better, it does not resolve my concerns about the paper's contribution. I therefore firmly maintain my original rating of reject.
Response to Reviewer h8y7
Thank you for the review. Numbers marked "post-review" come from a re-analysis of the stored records after we read the reviews. The revision prints them in Appendix A.3 (Tables 7 to 12). They are exploratory and change no registered result.
Why is Policy B not the proposed method? (Question 1) Your reading of Table 3 is right. At alpha=0.3, Policy B moves coverage from 38.3% to 55.3% on OS-Atlas and from 67.4% to 69.6% on UI-TARS, at executed risk 28.4% and 27.8% (Table 7). It still blocks every checker failure. Post-review, we found two limits. Its lower threshold admits certified proposals that err more often (197 at 65.0% on OS-Atlas, where its risk interval contains alpha). Calibrated on AndroidControl alone, Policy A certifies UI-TARS at alpha=0.1 and Policy B does not. The revised conclusion states that Policy B raises coverage at the three non-vacuous settings of the evaluated mixture, and that this does not establish that it is better under per-slice calibration or in live execution. Policy A stays in the paper as the configuration under study; we did not show that its requirement of a passing check has practical value.
Other predictors (Question 2). Post-review, we compared the gate with an agreement entropy (a surrogate for semantic entropy over our action equivalence) and mean sample confidence, both computed from the same samples (Table 8). The gate's AURC is lower than that of the agreement entropy by 0.0325 [0.0175, 0.0475] on AndroidControl and by 0.0346 [0.0147, 0.0538] on Mind2Web. Over 8 legs, its advantage excludes zero on 3 against the agreement entropy and on 8 against mean sample confidence, and it is worse on none. External calibrated gates are still not compared.
Alpha=0.30 for irreversible actions (Question 3). It is not acceptable for those actions, and the revised limitations say so. The lowest certified target is alpha=0.2 on OS-Atlas, at 8.7% coverage with executed step risk 3.8%. With the simultaneous bound on the pooled mixture and 952 anchors, no lower target is certified. Post-review, we estimated the anchors needed at the observed rate (Appendix A.3): OS-Atlas needs 1,950 at alpha=0.1 and 8,659 at alpha=0.05. Two other analyses reach alpha=0.1 but give no pooled guarantee: the exploratory fixed-sequence rule below (8.7% coverage on OS-Atlas) and calibration on AndroidControl alone, which certifies UI-TARS on AndroidControl only (43.7% coverage, executed risk 8.6%).
Treating the threshold sets as independent. The simultaneous bound does not assume independence: the union bound needs none, and the revised method section says so. It does ignore nesting, and your suggestion helps. Post-review, we tried a fixed-sequence rule that uses the nesting (Table 11). On the published split, coverage goes from 0.0% to 8.7% at OS-Atlas alpha=0.1, from 8.7% to 21.4% at alpha=0.2, from 38.3% to 41.7% at alpha=0.3, and from 67.4% to 74.5% at UI-TARS alpha=0.3. Its starting threshold depends on the calibration scores, and we have no proof that it keeps the error rate at delta, so the revision reports it as exploratory. On 200 resplits, under the reading that we registered (resplits with a non-empty certificate), anchor risk exceeds alpha in at most 10% at 5 of 6 settings and in 20.0% (3 of 15) at OS-Atlas alpha=0.05, where the published split is vacuous. Divided by all resplits (a post hoc reading), the largest rate is 8.0%. Step-weighted risk exceeds alpha in up to 54.2% of the non-empty resplits. The certified method stays the simultaneous bound.
Variation across splits. The certificate is marginal over the pooled mixture, and the paper states that the certified level does not transfer across splits. Post-review, per-slice calibration under Policy A is vacuous in 27 of 32 cells (Table 10), so these data cannot support a per-split guarantee.
Negative results and qualifiers. The revision narrows the stated contribution: the abstract and introduction now present the gate as the configuration under study in an offline diagnosis of what it certifies and what its veto costs, with ASK as a routing rule whose value we do not measure. The conclusion now states what Table 3 shows for Policy B, together with its limits. The qualifiers that remain state the scope of the claims (offline, step-level, marginal over the pooled mixture).