- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DecompBench contains 253 harmful tasks constructed from capability-based DAGs. It evaluates attacks using benign-looking subtasks, shared environment state, and cleared conversation history. Across seven agents, decomposition reduces refusals and increases attack success; GPT-5-mini rises from 0% to 68.25%.
Critical strengths
S1: The benchmark measures concrete harmful outcomes in stateful, multi-service environments; 74.9% of checkpoints are solely rule-based. S2: Large effects across seven models, independent guard checks, human agreement samples, and execution-failure analyses support the reported vulnerability.
Critical weaknesses
W1: Cross-benchmark refusal and success rates do not independently establish greater harmfulness, difficulty, or superiority of the construction principle. Task composition and technical failures differ across the compared datasets (Section 5.1; Appendix G). W2: Run variability is unevenly assessed: four frontier agents have one run, while Haiku and Qwen vary temperature across their three runs (Appendix E). Their reported variability therefore does not isolate repeatability under fixed settings. W3: Appendix C validates benignness per subtask, but does not report how many complete attacks have every subtask judged benign. This leaves uncertainty about attack success specifically within the subset satisfying C3 throughout.
Questions for authors
Q1: Are decomposer configurations, execution limits, and success/refusal scoring aligned between MT-AgentRisk and DecompBench in Table 5? Q2: Can paired task-level uncertainty intervals be reported for monolithic-versus-decomposed ASR, separately from variation across temperatures? Q3: Under each guard, what fraction of attacks have all subtasks labeled benign, and what is ASR within that subset?
Response to Reviewer WWBV (1/2): Subtask Benignness
Part 1 of response
Thank you for the thoughtful review and for recognizing the value of evaluating agent safety through what the agents actually accomplish. We appreciate your assessment that the consistency of the findings across models makes DeCompBench a useful resource for studying agent misuse.
Next, we address your questions below.
Q1: Subtask benignness
Thanks for the question. When decomposing the tasks and running them on an agent, we agree that benign-subtask files can still request a harmful action, but when running the decomposed subtasks, the attacker only cares about the benignness of subtasks as long as they are not refused by the agent. The low rates of refusal for our subtasks is evidence in itself that the subtasks were mostly considered benign. 196/253 (77%) of the tasks involve creating files, while the refusal rate is still very low, limited to a maximum of 16.60%.
Also, the dataset artifact released by us is the harmful monolithic tasks, and the users are free to use any decomposer of their choice to decompose the tasks: our decomposer (which uses mapping and wrapping operators) is just one instantiation, and a stronger attacker can use different operations that make the subtask files look even more benign.
Additionally, most of the files and scripts are created during the run of the previous subtasks, so it is not possible for the authors to make them benign by hand while the run is still going on, and hence the best we could do is use an LLM to automate making them benign-looking with our operators. The guards and human annotator evaluated the subtask instructions independently of the execution runs, without inspecting the referenced files or scripts.
We would like to clarify that the benign subtask isolation property is part of the harmful task creation methodology and it applies to the process before the harmful task construction. We will note in the paper that this is limited to the prompts.
...continued below