acceptodds
Under review as a conference paper at ICLR 2027

CAVEAT: Two-View Auditing with a Deterministic Veto Lowers GUI-Agent Verifier False Positives Below Recorded LLM Judges

Abstract

Models that read a recorded GUI-agent trajectory and predict task success accept narration as evidence. Most of a record audit’s false-positive reduction comes from a deterministic structural check, not from a second model view: of the expert-false test rows the view that reads the agent’s answer reports 67.7%, both views 58.1%, and the check 22.6% (78.6% of the drop), at 50.0% recall against the best recorded judge’s 93.5%. We present CAVEAT (Counterfactual Agent Verification via Evidence-only Auditing of Trajectories), the selective verifier that combines them: it reports success only when both reasoning-free views agree and the deterministic checker finds no structural contradiction. On the untouched test split the audit rule reports at 42.7% coverage with 81.6% precision (22.6% FPR), while the strongest measured LLM-judge, critic, reward-model, or single-view baseline reaches 73.4% precision and none gets below 35.5% FPR; the predeclared 10% risk certificate fails calibration (95% upper bound 65.4%), so the deployed policy abstains rather than overclaim. Because neither view reads agent reasoning, rewrites confined to reasoning cannot change a verdict; we prove this but do not measure it, since every implemented gaming attack also rewrites the terminal answer, and those attacks raise the gate’s false-positive rate on the pilot from 22.2% to 66.7%. The certificate fails partly by construction: the target needs 29 task-level reports and the rule produced 27. A live layer ran step ablation and state mutation on restored WebArena/VisualWebArena snapshots for a 30-trajectory design pilot (ablation returned a verdict on 10 of 22 replayable trajectories, mutation on none); an ablation-confirmed claim was expert-false, so we do not equate a verification record with truth. A 7B LoRA student reproduces 83.3%–88.9% of the frozen auditor’s view verdicts (3 seeds) at 6.04–6.42× its latency and fails its own certificate at every seed.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
GUI agents, trajectory verification, LLM-as-judge, selective verification, false positives, reasoning-free auditing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CZ23OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at automated checkers that read a recording of an AI agent using a website and decide whether the agent finished its task. These checkers are often fooled because they take the agent's own claims of success at face value. The authors build a stricter checker called CAVEAT. It says "success" only when two AI reviewers agree, one seeing the agent's final answer and one not, and a simple rule-based check finds nothing wrong in the record, such as the agent getting stuck in a loop, hitting an error, or giving an answer that never appeared on screen. Because neither reviewer reads the agent's private reasoning, rewriting that reasoning can't trick the system. The stricter checker wrongly approves failed tasks much less often than existing AI judges do. But it also rejects many tasks that really did succeed. Most of the improvement comes from the simple rule-based check, not the second AI reviewer, and an ordinary AI judge paired with that same check works about as well. The authors also set a strict reliability standard the system had to meet before being trusted. It fell short, so by design it declines to make any calls at all. A smaller distilled version and an early test of actively replaying agent actions also fell short. The overall message is that a basic structural check is the useful ingredient, at a real cost, and the system isn't yet reliable enough to deploy.

Critical strengths
  • I think the problem area is important and well motivated.
  • Some of the aspects around designing theory for abstaining on failures seems promising.
Critical weaknesses
  • Figure 1 is completely incomprehensible.
  • Figure 2 is fully AI generated. Looks very bad, given reviewer gives substantial time to read through these papers with no to little use of AI.
  • FIgure 4 not rendered properly.
  • I think the overall premise of the paper seems more suitable for a systems conference.
  • "RQ4 takeaway: rewrites confined to reasoning cannot move a verdict (Theorem 1; proved, not measured here); rewrites touching the terminal message move the operator gate’s worst-case FPR from 22.2% to 66.7% on the pilot, and both pre-registered operator-gate hypotheses fail." looks fully AI generated.
  • The useful component is four hand-written heuristics (loop at the end, error in the last three actions, missing target element, answer words not found on screen), tuned on a 30-trajectory pilot. A single existing judge with this check matches the full rule (p = 1.0). The two-view audit and the live intervention layer add nothing measurable on the holdout, and the live layer was never run on it.
  • Theorem 1 holds almost by construction, because the reasoning field is simply never given to the model. It assumes batch-invariant decoding, which wasn’t verified for the vLLM setup used. Every actual attack rewrote the final answer, which the checker does read. On the pilot, the gate’s false-positive rate rose from about 22% to 67% under attack. The “answer grounded on screen” predicate also looks easy to game by echoing on-screen text.
Questions for authors

Most of the paper is hard to parse and very AI sloppy. So i dont have any further valid questions to ask the authors. Further pls dont submit papers like this to flagship conferences without proper reading yourself the paper.

2 comments
AuthorsOpenReview ↗

Response to Reviewer CZ23

Thank you for noting that the problem is well motivated and that the theory for abstaining on failure is promising.

Figures and presentation. We redrew all six figures in the revision. Figure 2 shows what each consumer of a recorded trajectory reads. Figure 4, the certificate figure, is redrawn. The paper's AI use statement lists where generative AI tools were used, including figure preparation and editing the text. The authors reviewed all AI-assisted work and take responsibility for the final content. The RQ4 takeaway states in one sentence what Theorem 1 covers and what the pilot measured.

The useful component and the live layer. The paper's central finding is that the deterministic check drives the false-positive reduction (78.6% of the drop). The check is four fixed record predicates (Appendix E). The test split excludes every pilot task id. Three pilot tasks return under alternate ids, and the headline holds without them and the 11 other affected rows (22.2% against 66.7%; Appendix H). No threshold, prompt or teacher deployment decision uses a test-split trajectory or label, with one disclosed exception that favours the baseline family; the student's certificate is computed on test representatives (Section 4). On that split the rule's FPR is 22.6% against 67.7% for the best recorded judge (McNemar p=0.0026). A pre-registered evaluation on unused tasks of two other environments agrees with your reading (Appendix I): at an equal number of reports the rule's FPR is 13.0% against an expected 29.5% for the judge with the checker thinned at random, and the interval of the difference, [-32.2, +0.3] points, contains zero. The revision therefore recommends one judge gated by the checker, whose certificate is not yet measured, and keeps the two views as the analysed variant for which the certificate is measured and Theorem 1 proved. The semantic view removes expert-false rows selectively (odds ratio 6.90, 95% CI [1.71, 32.70], p=0.0019; Section 6). The live intervention layer is a 30-trajectory design pilot, reported separately. The holdout estimates use the offline rule only.

Theorem 1. The theorem is an exact structural guarantee. The reasoning field is not an input to either view or to the checker. A rewrite of that field leaves every prompt and the checker's input byte-identical (Appendix A). The theorem states its premise: deterministic, batch-invariant auditor decoding. We did not verify that premise for our batched vLLM decoding, and Section 4 says so. The paper labels the result "proved, not measured here" in RQ4.

Attacks that rewrite the terminal answer. Theorem 1 covers rewrites confined to reasoning. The abstract and Section 3.2 state this scope. An answer that echoes on-screen text changes the terminal message, which the rule reads, so it lies outside the theorem. You are right that the grounding predicate is easy to satisfy: in a diagnostic on the 96 test records, an answer replaced by one token copied from the screen passes it on all 96 (results/rebuttal_20261107/readout.json in the supplementary package). The predicate fires only when no content token of the answer appears in the recorded observations, so it cannot exclude an echo. Section 5 (RQ4, Figure 6) reports the pilot results for the four attack families on the 30-record design pilot, for both the re-judge and the operator gate.

Venue. We submitted under alignment, fairness, safety, privacy and societal considerations. The paper studies when a verifier of agent trajectories may report success. It certifies the risk of its reports and abstains when the certificate fails. The experiments use a public benchmark, pinned at an immutable revision, and open-weight models.

I have read the author responses, but i am not satisfied with the initial stage of the draft given how it was submitted. I wish to keep my evaluation.

Reviewer_qjR5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether GUI-agent success verification can be improved by combining two model views with a deterministic structural veto. The veto checks observable trajectory features, while the model views assess different portions of the available evidence. Evaluation uses expert-labeled trajectories drawn from episodes accepted by a recorded verifier, with separate training, calibration, and test partitions. On the 96-trajectory test set, the complete system achieves a 22.6% false-positive rate, substantially below a strong recorded judge but with considerably lower recall and coverage. Ablations attribute much of the reduction to the structural veto and provide limited evidence for an additional benefit from the second view. The paper also studies selective-risk certification, finding that the available calibration data do not support the specified deployment target.

Critical strengths
  1. The paper carefully identifies what actually drives the observed improvement. Rather than attributing the lower false-positive rate to the full architecture, the ablations show that a simple structural veto accounts for much of the reduction. This is a useful corrective to claims that additional model views necessarily improve verification.

  2. The evaluation reports the costs of conservative verification. False-positive reductions are accompanied by explicit recall and coverage measurements, paired comparisons, and task-clustered uncertainty estimates. The calibration analysis likewise makes clear that favorable test-set performance does not establish the requested deployment-risk guarantee.

Critical weaknesses
  1. The practical advantage of the structural veto over simply accepting fewer trajectories remains insufficiently resolved. The lower false-positive rate is clear against a more permissive recorded judge, but that comparison also involves a substantial loss of recall and coverage. The coverage-matched random-thinning analysis has a confidence interval spanning zero, and the test of the veto’s error selectivity is inconclusive. These results do not establish that the veto lacks value; they leave uncertain whether its structural checks provide a reliable advantage beyond conservatism. That uncertainty limits the paper’s principal positive contribution.

  2. The empirical basis is narrow for establishing a reusable verifier-design result. The test set contains 96 trajectories from 56 tasks, including 31 negative examples, and is conditioned on acceptance by one recorded verifier. This supports conclusions about the evaluated subset, but provides limited evidence about performance across a broader deployment distribution. The calibration shortfall is informative for this sample, although it does not establish a general limitation of the approach or certification procedure.

Questions for authors
  1. Clarification: Which reported analysis provides the strongest evidence that the structural veto preferentially removes incorrect acceptances, after accounting for coverage? How should that evidence be interpreted given the inconclusive coverage-matched and selectivity analyses?

  2. Clarification: Can performance be stratified by veto rule and task family, including the corresponding true-positive losses, to determine whether the aggregate result is driven by a small number of recurring trajectory patterns?

1 comment
AuthorsOpenReview ↗

Response to Reviewer qjR5

Thank you for the careful review and for describing what the ablations show.

Which analysis gives the strongest evidence that the veto removes incorrect acceptances, after coverage? Selectivity compares the rate at which a step removes expert-false rows with the rate at which it removes expert-true rows, so removing more rows does not by itself raise it. Our plan fixed a strength score for each of its 14 estimands: the absolute point estimate over the half-width of its interval. The strongest is the whole rule on the best recorded judge's accepted rows. It withholds 17 of 21 expert-false rows (81.0%) and 29 of 58 expert-true rows (50.0%). The difference is 31.0 points, task-cluster 95% interval [7.3, 53.5]. On the same records, the structural checker applied to the best recorded judge's accepted rows removes 15 of 21 expert-false rows (71.4%) and 24 of 58 expert-true rows (41.4%). The difference is 30.0 points, task-cluster 95% interval [4.2, 55.4]. The whole rule on all 96 test rows removes 77.4% of expert-false rows and 50.0% of expert-true rows. The difference is 27.4 points, interval [4.9, 49.1]. These are three of the 14 intervals (clustered by original task id), which we specified in a plan committed before the scripts and results of this task-cluster re-analysis. Four exclude zero (no multiplicity adjustment), all selectivity differences in favor of the rule. The six FPR differences at matched coverage do not exclude zero. None of the 14 excludes zero against the rule. The paper also claims two gaps in Section 6 (Figure 5): the headline FPR gap and the veto's own step. They are 5.0 and 4.0 times the task-cluster noise floor.

Stratification and recurring patterns. Appendix H now lists every stratum and the counts per veto rule. On the stored records, the veto removes 11 expert-false and 23 expert-true rows from the 75 rows that both views accept. By veto rule: the loop predicate removes 5 expert-false and 1 expert-true rows, the execution-error predicate removes 6 and 22, and the other two predicates do not fire on the 96 test rows. This split is exploratory. By task: the 11 rows lie in 9 tasks. The largest task holds 27.3% of them, and the three largest hold 45.5%. The 23 rows lie in 13 tasks. Omitting any one of the 56 task ids leaves the veto's expert-false removals between 8 and 11 and the rule's FPR difference to the best recorded judge between -51.7 and -42.9 points (full sample -45.2). Omitting any one benchmark, agent, site or evaluation-type stratum leaves them between 5 and 11 and between -58.3 and -31.8 points. No omission changes the sign of that difference. The strata hold few tasks, so a stratum without a visible effect is not evidence of no effect. The four veto predicates are defined in Section 3.2 and Appendix E.

Basis of the evidence. The test split holds 96 trajectories from 56 task ids, and 31 of them are expert-false. Task ids went to splits by salted hash, and every pilot task id was excluded. After the reviews we found that the dataset gives a VisualWebArena task and its resized copy different ids, so 16 underlying tasks span two splits and 3 pilot tasks reach the test split. The revision states this in Section 4 and repeats the headline without the 14 affected test rows (Appendix H, exploratory). On the remaining 82 rows from 41 tasks, the rule's FPR is 22.2% against 66.7% for the best recorded judge: a difference of -44.4 points, task-cluster 95% interval [-69.3, -18.5], McNemar p=0.0042. No cohort, split or certificate-representative choice reads the expert labels; verdicts are scored against them only in the aggregator (Section 4). The paper states every claim for the primary public judge's accept set (Section 7). The headline test runs at task level: the exact sign-flip test gives p=0.0019 and the cluster-bootstrap interval for the FPR difference is [-70.0, -19.4] points ([-70.0, -20.5] with the merged task ids). The certificate is a statement about the evaluated distribution. Theorem 2 gives an exact bound for i.i.d. tasks, and the deployed policy abstains when the bound fails. A pre-registered evaluation adds 111 trajectories from unused tasks of two other environments (Appendix I). There the rule's FPR is 13.0% at 45.0% coverage, and at an equal number of reports the judge with the checker is not separated from the rule. On the test split, at their own reporting rates, the best judge with the checker (19.4% FPR) and the rule (22.6%) are not separated either (McNemar p=1.000).

Reviewer_AqvPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The main contribution from this paper is a study of whether checking a GUI agent’s trajectory can make judgments of task success more reliable. CAVEAT takes two model “views,” where one reads the agent’s final answer and one that doesn’t read it. Both omit the agent’s reasoning, and a structural checker can reject a success report based on problems (i.e. execution errors, missing action targets). Then, they use a calibration gate that allows recording if the rule passes a predeclared error target, and on 96 tests they selected from a public judge’s accept set, the false-positive rate falls from 67.7% to 22.6% but recall falls from 93.5% to 50.0%. Most of the reduction just comes from the checker, and the biggest weakness is that a single judge with the same checker already does comparably. The calibration gate also fails so the deployed rule has no successes.

Critical strengths
  • The controls are useful because they let us see where the improvement actually comes from but also show the main weakness of the paper i.e. the checker-alone and single-judge-plus-checker ablations do indeed separate the contribution of a filter from that of the second model view. Could help evaluate a more complicated verifier.
  • They disclose the limitations.
Critical weaknesses
  • I don’t see what the full method adds over a much simpler verifier. A single judge with the same checker has a 19.4% false-positive rate and 54.8% recall, compared with CAVEAT’s 22.6% and 50.0%, and the paired test doesn’t differentiate them. From my understanding, the second model view only removes three additional false positives before the checker is applied which means that the extra model view isn’t needed for the main result. I don’t think the theoretical guarantees apply here since they exclude reasoning from the inputs and just apply a standard binomial bound.
  • The practical benefit of the reduction in false positives is also not well established. Recall falls from 93.5% to 50.0%, and coverage falls from 84.4% to 42.7% (Table 1). When the judge’s reports are randomly thinned to a comparable coverage, the estimated improvement is 11.7 percentage points, but its confidence interval has no improvement. So this generally doesn’t resolve the main question of whether the checker makes better selections.
  • Heavy-LLM use and unparsable figures made it difficult for me to read, see Figure 1 (overlapping data, confidence bars, text, etc. - cannot read). Appendix H says 25 tasks but has 19 rows, and some phrases like “the paper’s FP numbers” which points to a “rejudge” and “live gate” (p.17) which seems like placeholders that weren’t updated before submission. I still went through and read as much as I could but many parts were difficult to parse.
Questions for authors
  • What does the second model view add over a single judge with the same structural checker?
  • Does the checker improve success judgments when the reporting rates are matched?
1 comment
AuthorsOpenReview ↗

Response to Reviewer AqvP

Thank you for the detailed review and for noting that the controls show where the improvement comes from.

What the second view adds over a single judge with the same checker. We ran this comparison on new tasks. Before computing any verdict we committed a protocol with one primary contrast and a reading rule (Appendix I). The pool is every WorkArena and AssistantBench trajectory that the public judge accepts: 111 trajectories on 53 tasks, 23 of them expert-false, from environments that no earlier step used. The views and the checker were frozen. At 50 reports for each rule, the rule's FPR is 13.0%; the Table 1 row 2 judge with the checker, thinned at random to 50 reports, has an expected FPR of 29.5%. The difference is -16.4 points, task-cluster 95% interval [-32.2, +0.3]. The interval contains zero, so by the pre-registered reading the second view is not shown to add value over the judge with the checker. The revision therefore recommends one recorded judge gated by the checker (Section 3.2). Its certificate is not yet measured; the two views remain the analysed variant for which the paper measures the certificate and proves Theorem 1 under deterministic, batch-invariant decoding. On the test split the deterministic checker supplies most of the false-positive reduction, 35.5 of the 45.2 points, or 78.6% (Section 6), and the judge plus checker reaches 19.4% FPR at 54.8% recall and the rule 22.6% at 50.0%, both far below the best recorded judge at 67.7%.

Practical benefit of the false-positive reduction. The FPR falls from 67.7% to 22.6% on the same 31 expert-false test rows. This gap is 45.2 points, or 5.0 times the task-cluster noise floor. The paired exact McNemar test gives p=0.0026. The task-level exact sign-flip test gives p=0.0019. The task-cluster bootstrap interval for the difference is [-70.0, -19.4] points. The recall and coverage values in Table 1 are the cost of reporting only when both views and the checker agree. The paper states them next to the FPR.

Does the checker help when reporting rates are matched? Selectivity compares how often a step removes expert-false rows and expert-true rows. Under random removal its expected value is zero at any removal volume. On the stored records, the rule removes 24 of 31 expert-false test rows (77.4%) and 31 of 62 expert-true rows (50.0%). The difference is 27.4 points, task-cluster 95% interval [4.9, 49.1]. The structural checker applied to the best recorded judge's accepted rows removes 15 of 21 expert-false rows (71.4%) and 24 of 58 expert-true rows (41.4%). The difference is 30.0 points, interval [4.2, 55.4]. These two intervals (clustered by original task id) belong to a set of 14 that we specified in a plan committed before the scripts and results of this task-cluster re-analysis. Four of the 14 exclude zero, with no multiplicity adjustment. All four are selectivity differences in favor of the rule. The six FPR differences at matched coverage do not exclude zero. None of the 14 excludes zero against the rule.

Guarantees. Theorem 1 gives an exact invariance for the reasoning channel, and the rule reads no reasoning. Theorem 2 gives an exact one-sided binomial bound on the task-level selective risk, with one preselected representative per task. The bound decides whether the rule may report (Section 6).

Presentation. Fixed in the revision: Appendix H now reads "25 tasks holding an expert-false row, of which 19 hold a discordant row (20 discordant rows in all)". Fixed in the revision: the Appendix F table caption now says that the main-text false-positive rates come from the expert-labelled test split (Section 5) and the pilot's rates from Appendix G. We also redrew all six figures.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Title change in the revision

Title change. The revised PDF is titled "CAVEAT: A Deterministic Structural Veto Rather Than a Second Model View Lowers GUI-Agent Verifier False Positives, at a Recall Cost". The submitted title was "CAVEAT: Two-View Auditing with a Deterministic Veto Lowers GUI-Agent Verifier False Positives Below Recorded LLM Judges". The new title states the revised finding: the structural checker, not the second model view, gives most of the false-positive reduction, and the reduction costs recall.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.