- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces SEAD, a unified framework that models attack and defense in tool-using agents as a partially observed state-control problem, emphasizing how cumulative environment changes can lead to harmful outcomes. Building on this formulation, the authors propose DART, an adaptive attack method that uses execution feedback and tree search to construct multi-step attacks, and SAGE, a pre-execution defense that actively investigates environment states before allowing or blocking tool actions.
Experiments cover 187 harmful tasks across four target models, with additional offline and online defense evaluations. DART consistently outperforms existing attack baselines, while SAGE reduces executable attack success from 48.0% to 4.0% in online evaluation and preserves 95.79% of benign trajectories in offline evaluation. The authors also report encouraging generalization to held-out tool domains and different attack methods.
Critical strengths
-
A useful state-based perspective on agent safety. The paper makes a convincing case that harmfulness should be assessed through actual environment transitions rather than isolated tool calls. The unified formulation of attack and defense is intuitive and practically relevant.
-
Strong empirical results with executable verification. The use of replayable environments and executable success checks is a meaningful strength. In particular, SAGE reduces online hard ASR from 48.0% to 4.0%, with encouraging results on held-out domains.
Critical weaknesses
-
The attack comparisons are not sufficiently controlled. DART uses task-specific executable verifier progress during search, and its feedback critic receives completion criteria (Sec. 4.1, Appendix F.3). The competing methods also operate under different search budgets. Without matched feedback access and computational budgets, it is difficult to attribute DART's gains to the proposed search strategy.
-
The contribution of state investigation is not adequately isolated. SAGE lacks a controlled ablation without environment queries. Moreover, 58.18% of harmful trajectories are blocked before the annotated closure, and online benign task completion is not evaluated. These limitations weaken the claims about precise interception and safety–utility balance.
Questions for authors
-
How much of DART's improvement remains under matched execution budgets and without privileged verifier-progress feedback?
-
How does SAGE perform without read-only environment queries, keeping the backbone and training setup unchanged? Could the authors also clarify how the harmful-closure annotations are validated and report online benign task completion?
Thanks for the detail rebuttal. I will maintain the current positive score.