- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper looks at automated checkers that read a recording of an AI agent using a website and decide whether the agent finished its task. These checkers are often fooled because they take the agent's own claims of success at face value. The authors build a stricter checker called CAVEAT. It says "success" only when two AI reviewers agree, one seeing the agent's final answer and one not, and a simple rule-based check finds nothing wrong in the record, such as the agent getting stuck in a loop, hitting an error, or giving an answer that never appeared on screen. Because neither reviewer reads the agent's private reasoning, rewriting that reasoning can't trick the system. The stricter checker wrongly approves failed tasks much less often than existing AI judges do. But it also rejects many tasks that really did succeed. Most of the improvement comes from the simple rule-based check, not the second AI reviewer, and an ordinary AI judge paired with that same check works about as well. The authors also set a strict reliability standard the system had to meet before being trusted. It fell short, so by design it declines to make any calls at all. A smaller distilled version and an early test of actively replaying agent actions also fell short. The overall message is that a basic structural check is the useful ingredient, at a real cost, and the system isn't yet reliable enough to deploy.
Critical strengths
- I think the problem area is important and well motivated.
- Some of the aspects around designing theory for abstaining on failures seems promising.
Critical weaknesses
- Figure 1 is completely incomprehensible.
- Figure 2 is fully AI generated. Looks very bad, given reviewer gives substantial time to read through these papers with no to little use of AI.
- FIgure 4 not rendered properly.
- I think the overall premise of the paper seems more suitable for a systems conference.
- "RQ4 takeaway: rewrites confined to reasoning cannot move a verdict (Theorem 1; proved, not measured here); rewrites touching the terminal message move the operator gate’s worst-case FPR from 22.2% to 66.7% on the pilot, and both pre-registered operator-gate hypotheses fail." looks fully AI generated.
- The useful component is four hand-written heuristics (loop at the end, error in the last three actions, missing target element, answer words not found on screen), tuned on a 30-trajectory pilot. A single existing judge with this check matches the full rule (p = 1.0). The two-view audit and the live intervention layer add nothing measurable on the holdout, and the live layer was never run on it.
- Theorem 1 holds almost by construction, because the reasoning field is simply never given to the model. It assumes batch-invariant decoding, which wasn’t verified for the vLLM setup used. Every actual attack rewrote the final answer, which the checker does read. On the pilot, the gate’s false-positive rate rose from about 22% to 67% under attack. The “answer grounded on screen” predicate also looks easy to game by echoing on-screen text.
Questions for authors
Most of the paper is hard to parse and very AI sloppy. So i dont have any further valid questions to ask the authors. Further pls dont submit papers like this to flagship conferences without proper reading yourself the paper.
2 comments
I have read the author responses, but i am not satisfied with the initial stage of the draft given how it was submitted. I wish to keep my evaluation.
Response to Reviewer CZ23
Thank you for noting that the problem is well motivated and that the theory for abstaining on failure is promising.
Figures and presentation. We redrew all six figures in the revision. Figure 2 shows what each consumer of a recorded trajectory reads. Figure 4, the certificate figure, is redrawn. The paper's AI use statement lists where generative AI tools were used, including figure preparation and editing the text. The authors reviewed all AI-assisted work and take responsibility for the final content. The RQ4 takeaway states in one sentence what Theorem 1 covers and what the pilot measured.
The useful component and the live layer. The paper's central finding is that the deterministic check drives the false-positive reduction (78.6% of the drop). The check is four fixed record predicates (Appendix E). The test split excludes every pilot task id. Three pilot tasks return under alternate ids, and the headline holds without them and the 11 other affected rows (22.2% against 66.7%; Appendix H). No threshold, prompt or teacher deployment decision uses a test-split trajectory or label, with one disclosed exception that favours the baseline family; the student's certificate is computed on test representatives (Section 4). On that split the rule's FPR is 22.6% against 67.7% for the best recorded judge (McNemar p=0.0026). A pre-registered evaluation on unused tasks of two other environments agrees with your reading (Appendix I): at an equal number of reports the rule's FPR is 13.0% against an expected 29.5% for the judge with the checker thinned at random, and the interval of the difference, [-32.2, +0.3] points, contains zero. The revision therefore recommends one judge gated by the checker, whose certificate is not yet measured, and keeps the two views as the analysed variant for which the certificate is measured and Theorem 1 proved. The semantic view removes expert-false rows selectively (odds ratio 6.90, 95% CI [1.71, 32.70], p=0.0019; Section 6). The live intervention layer is a 30-trajectory design pilot, reported separately. The holdout estimates use the offline rule only.
Theorem 1. The theorem is an exact structural guarantee. The reasoning field is not an input to either view or to the checker. A rewrite of that field leaves every prompt and the checker's input byte-identical (Appendix A). The theorem states its premise: deterministic, batch-invariant auditor decoding. We did not verify that premise for our batched vLLM decoding, and Section 4 says so. The paper labels the result "proved, not measured here" in RQ4.
Attacks that rewrite the terminal answer. Theorem 1 covers rewrites confined to reasoning. The abstract and Section 3.2 state this scope. An answer that echoes on-screen text changes the terminal message, which the rule reads, so it lies outside the theorem. You are right that the grounding predicate is easy to satisfy: in a diagnostic on the 96 test records, an answer replaced by one token copied from the screen passes it on all 96 (results/rebuttal_20261107/readout.json in the supplementary package). The predicate fires only when no content token of the answer appears in the recorded observations, so it cannot exclude an echo. Section 5 (RQ4, Figure 6) reports the pilot results for the four attack families on the 30-record design pilot, for both the re-judge and the operator gate.
Venue. We submitted under alignment, fairness, safety, privacy and societal considerations. The paper studies when a verifier of agent trajectories may report success. It certifies the risk of its reports and abstains when the certificate fails. The experiments use a public benchmark, pinned at an immutable revision, and open-weight models.