- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper evaluates whether LLM agents voluntarily recheck authorization when permission changes between preparation and an irreversible action. Its five matched conditions distinguish stable authority, silent revocation, supplied revocation evidence, and explicit requirements to reread authorization. Deterministic tool traces separately measure evidence acquisition, unauthorized effects, successful task handling, and execution failures. The evaluation covers five model endpoints, 30 human-reviewed synthetic tasks, and three repetitions per condition. GPT-5.4 completes 89/90 stable-authority trials but produces unauthorized effects in 40/90 silent-revocation trials; supplied revocation evidence yields successful safe recovery in 90/90 trials. Across endpoints, all 94 observed silent-revocation violations occur without a fresh read, although the paper correctly treats this as an association rather than proof of a causal effect.
Critical strengths
-
The experimental design separates meaningfully different authorization behaviors. The five conditions distinguish voluntarily acquiring current evidence, using supplied evidence, following an explicit reread instruction, and completing legitimate tasks. The GPT results demonstrate why these distinctions matter: high stable-task competence and successful use of revealed evidence coexist with frequent failures to revalidate voluntarily. This provides a useful diagnostic beyond aggregate violation rates.
-
The evaluation carefully distinguishes safety from execution failure. Invalid calls, abandonment, and exhausted budgets do not receive successful-recovery credit, and the appendix reports their concrete causes. Task-level bootstrapping also respects the distinction between 30 independent tasks and repeated trajectories. This reporting materially strengthens confidence in the central findings and prevents unreliable endpoints from appearing safe merely because they fail to act.
Critical weaknesses
-
Scaffold compatibility substantially affects some behavioral comparisons. The one-tool-call rule accounts for 141 of Claude’s 173 invalid executions, while 64 of Kimi’s 77 unsuccessful stable-required trajectories contain a server-reported length termination under the 512-token response limit. These observations limit how confidently recovery differences and required-read utility losses can be attributed to authorization behavior, even though the authors appropriately acknowledge the confounding. Interface and response-budget sensitivity controls would strengthen the interpretation.
-
The evidence covers a narrow model of dynamic authorization. The benchmark contains one synthetic task per mechanism–effect cell, uses an exact trusted ledger, and schedules a single atomic authorization change after preparation; no further revocation occurs between a fresh read and commit. "Safe recovery" also means grounded denial, without requiring cleanup or an alternative authorized workflow. These choices provide strong experimental control but limit conclusions about ongoing authorization changes and realistic recovery, so broader workflow validation would improve the contribution’s practical reach.
Questions for authors
-
Was the one-tool-call restriction enforced through provider API settings or checked only after generation? Given Claude’s 141 multiple-call failures, this distinction materially affects the interpretation of its results.
-
Do the saved traces show how many revealed/required-condition failures occur after receiving current evidence, versus before reaching the relevant transition? Separating these would sharpen the distinction between evidence-use failures and workflow failures.
-
For four-step preparation tasks, how much of the eight-request budget remains for revalidation, committing, and recording the final decision? Please clarify whether the reference-policy validation also checks this execution-budget margin.