- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors present ClawTrojan, a benchmark for multi-step trojan attacks on agentic harnesses. Along side it they also introduce DASGuard, which is a defense system for this exact scenario. They show that commonly used methods for protecting agentic harnesses fall short on their attack benchmark, while their defensive technique prevails.
Critical strengths
For me, this paper addresses the very important and relevant issue of agent safety and security. The ClawTrojan dataset highlights a major blind-spot in current models and protections, especially with the low performance reported in Table 3. Furthermore, the DASGuard method presented shows very strong and encouraging results, however see Weaknesses. The ablation work in Table 6 is also very interesting to help isolate which components are the most impactful.
Critical weaknesses
My main concern for this paper is the co-development of a test dataset (ClawTrojan) and a defense mechanism (DASGuard) tuned to this dataset. This is particularly evident by the lack of test of DASGuard on other datasets. ClawTrojan also has a very high class imbalance, with comparatively few clean/control/borderline samples.
Multiple practical details are unclear for me:
- The precise attack and defense conditions are not clear to me. Is ClawTrojan focused on code contributors in projects? Can it also target external black-box agents? How is DASGuard integrated? Do you need full harness access (so limited to open source harnesses), or would it also work on proprietary harnesses?
- How is the DASGuard threshold established?
- It is not clear what base model was used when testing all the defense mechanisms in Table 3, and how the change of model impacts the performance of DASGuard and others. Although GPT 5.4 is stated as the base model used, I assume this is the one used for the defenses? However the table contains 5.6 and 5.6 is also reused in Table 4. A consistent model would improve the quality of the comparisons
Presentation: Generally understandable, but some sections could benefit from clearer writing (Sec. 4.2, 5) as they are not always easy to follow. A lot of notation and symbols are introduced in the equations but have little to no reuse. Also, Figure 1 has contradictory numbering and arrow orderings.
Section 5.1: is there a word missing? "records [how] that content is derived" (L.269)
Questions for authors
-
The main differentiation factor of ClawTrojan with existing benchmarks presented in Table 2 is Dialog, especially with AgentDojo. It is not clear to me how this category nuances from "Step chain" described as multi-step processes. If the authors could explicitate this please?
-
The authors distinguish "partial verdicts", where the final attack is blocked but unsafe residue remains. I think this is a direction that needs to be explored more when presenting a multi-step attack and defense setting. What remains? How dangerous is it? how easy is it to reuse? etc.
-
As mentioned above, was DASGuard tested on other datasets?
Smaller concerns:
- Could you provide a point of comparison for the values in Table 5, such as the results from CaMeL please?
- How is time translated to cost? I would like to see a cost/performance graph. I am also not clear on the time column in Table 3, if it represents additional run time or total time? (which would surprise me as raw-agents times are usually longer)