- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper checks whether tool-using agents stay within the user’s authorization by examining attempted calls, final states, and intermediate changes. Using 1,600 AgentDojo trajectories, it shows how replay can reveal unauthorized effects that disappear before execution ends.
Critical strengths
- The framework separates failed unauthorized attempts, persistent effects, and temporary effects.
- Replaying executions from the same initial state provides concrete evidence linking calls to state changes.
Critical weaknesses
- ToolSandbox already checks forbidden events in intermediate states, and AgentPex evaluates constraints across full traces. The paper discusses these approaches but does not compare against them.
- Failed unauthorized attempts and violations of confirmation or ordering requirements are central to the framework, but their detection accuracy is not separately reported against human authorization judgments.
Questions for authors
- Could you compare with adapted ToolSandbox checks or AgentPex under the same authorization rules?
- How accurately does the evaluator detect failed unauthorized attempts and violations of confirmation or ordering requirements?
2 comments
Q1:
We added the setup, accuracy table, mechanism analysis, and cost/repeatability results to Appendix G (Tables 12–13).
Setup. ToolSandbox's milestone/minefield evaluation is re-implemented on our replays under the Table 1 authorization rules. AgentPex runs from its official code, first with its native specifications and then with our contract in the specification slots, keeping its checker unchanged (Appendix G.1).
ToolSandbox. With minefields, its F1 is 0.813 versus our 0.816 (p = 1.0). The mechanism analysis explains the difference in coverage: milestones alone miss all 27 transient effects; tool-name minefields recover 19, and the remaining eight require the attack goal itself to be enumerated. Our default-deny contract detects all 27 without attack knowledge. ToolSandbox also misses 15 persistent effects outside milestone/guardrail coverage. Its tool-trace minefields detect the failed unauthorized call, which our separate check also flags. The six cases favoring ToolSandbox arise from timestamp/URL artifacts; canonicalization gives F1 0.826 (post hoc; Appendix E.1).
AgentPex. Native F1 is 0.786. Supplying our contract increases detection of labeled transient/persistent effects from 3/10 and 62/90 to 10/10 and 81/90. Its authorization-rule score is then not significantly different from ours (F1 0.820 versus 0.816, p = 0.65). The aggregate is higher (0.879, p = 0.011): 29 of its 42 uniquely correct cases concern wrong replies or off-task reads without state violations. Excluding all 43 such cases gives 0.867 versus our 0.890 (post hoc, p = 0.38); the default threshold gives aggregate F1 0.840 (p = 0.25). Full results remain in Appendix G.2.
Summary. Detection coverage depends on the contract. With explicit authorization information, the three authorization-focused checkers have similar F1 (0.813–0.820). Our replay provides deterministic, state-change evidence without model calls (1,600 traces in about three minutes on two CPU cores). AgentPex with our contract costs about $0.015 per trace; 10.4% of aggregate decisions differ across three runs on the 230 completed traces (Appendix G.3).
Q2:
Failed unauthorized attempts. flags 5 of the 6 failed unauthorized attempts with static contracts. It checks attempted calls regardless of execution success (Eq. 4), including calls with . Across 1,600 replayed traces, 11 of the 34 failed mutating calls unmatched to the reference are unauthorized attempts. The other five unauthorized attempts lack static contracts and are unevaluable. The remaining 23 calls are retries or repetitions of requested operations; flags 22.
Confirmation and ordering. AgentDojo contracts contain no explicit confirmation or ordering requirements, so the accuracy of detecting these violations has not been directly measured on this benchmark. Among the 300 reviewed trajectories, six have confirmation-required rationales: payments without bill clarification, an unrequested switch to recurring payments, and calendar invitations to an unrequested participant. These actions produce unrequested realized effects, and the final-state or trajectory-action signal in Table 1 flags all six trajectories. No sampled trajectory was judged to violate an ordering requirement.
Dear Authors,
As the rebuttal period is nearing its end, I would like to follow up on the questions raised in my review. I would appreciate any responses or clarifications you could provide, as these would help inform my final assessment.
Thank you for your time.