- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ClashBench, a benchmark for measuring risks of agents overstepping and/or preempting resources in use by other agents, comprising of 268 scenarios. The authors benchmark various SOTA LLM agents paired with various agentic harnesses, finding non-trivial amounts of preemption that does not disappear with simple prompts not to preempt other tasks, and that often is not mentioned in the summary of the execution.
Critical strengths
This paper is great and should be accepted (pending a small human validation, see weaknesses)! The benchmark is well-designed, the experiments and analyses are interesting, and it'll make a nice contribution to the field.
Critical weaknesses
The paper would be a clear accept to me, if authors can address the following:
- L248: Human validation is needed for the outcome grading, in order to assert eval validity. Doing it on a sample (e.g., 30-50 trajectories) would be enough. Same with L382 (conflict recognition, user reporting, and user deferral judge).
Additionally, some other suggestions for improvement:
- L190: It would be nice to have a bit more explanation on how the group conflict categories were devised? Was this theory-driven, data-driven, or some combination of both?
- Missing related works: https://arxiv.org/abs/2507.06134, https://arxiv.org/abs/2602.13379, http://arxiv.org/abs/2409.16427, https://aclanthology.org/2025.naacl-long.595/, https://www.anthropic.com/research/agentic-misalignment
- L161: I'm not 100% certain that the right behavior is to ask the user; perhaps waiting is an appropriate solution as well. I wish authors could add a bit more discussion on how agents should handle these conflicts of resources.
Post rebuttal: Authors addressed my comments, so raising my score 3 -> 4
Questions for authors
N/A
4 comments
Official Comment by Authors 2/2
W3: Missing related works.
We thank the reviewer for these valuable references. We will incorporate all five suggested studies into Related Work and clarify their differences from ClashBench.
Specifically, OpenAgentSafety [3] evaluates unsafe behaviors in realistic multi-user, tool-using tasks; MT-AgentRisk [4] investigates multi-turn agent safety risks and proposes the ToolShield defense; and HAICOSYSTEM [5] studies safety risks in simulated human-agent interactions. Additionally, AI-LieDar [6] examines deception arising from conflicts between task utility and truthfulness, while Agentic Misalignment [7] demonstrates harmful agent behaviors under goal conflicts and replacement threats.
Unlike these studies, ClashBench specifically investigates destructive resource preemption under benign user requests, measuring whether an agent completes its task by disrupting an existing task through executable environments and independent outcome graders. We will make these distinctions explicit in the revised Related Work section.
W4: Handling resource conflicts (L161).
Thank you for this insightful suggestion. When encountering resource conflicts, an agent generally has several options: (1) waiting for the resource to become available, (2) seeking alternative resources, (3) asking the user for guidance, or (4) directly preempting the occupied resource.
The appropriate choice depends on the context. Waiting may be suitable when the conflict is temporary, while alternative resources may not always be available. Among these options, consulting the user is often a preferable strategy when resolving the conflict requires trade-offs between competing tasks, as it preserves user autonomy and avoids unauthorized disruption of existing tasks. In contrast, unilateral destructive preemption is precisely the unsafe behavior that ClashBench aims to evaluate.
References
[1] Hindman, Benjamin, et al. "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center." NSDI, 2011.
[2] Ghodsi, Ali, et al. "Dominant Resource Fairness: Fair Allocation of Multiple Resource Types." NSDI, 2011.
[3] Vijayvargiya, Sanidhya, et al. "OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety." arXiv:2507.06134, 2025. https://arxiv.org/abs/2507.06134
[4] Li, Xu, et al. "Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents." arXiv:2602.13379, 2026. https://arxiv.org/abs/2602.13379
[5] Zhou, Xuhui, et al. "HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions." arXiv:2409.16427, 2024. https://arxiv.org/abs/2409.16427
[6] Su, Zhe, et al. "AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents." NAACL, 2025. https://aclanthology.org/2025.naacl-long.595/
[7] Anthropic. "Agentic Misalignment: How LLMs Could Be Insider Threats." 2025. https://www.anthropic.com/research/agentic-misalignment
Thanks :) Raising my score!
Dear Reviewer y3Ya,
Thank you for the encouraging review and for raising your score! We have added the related work you suggested, and your request to validate the graders against human labels led to the extended audit, which we have reported in the revised paper.
Best regards, Authors of Submission 21123
Official Comment by Authors 1/2
Thank you for your constructive feedback and for recognizing the value of our work on ClashBench.
W1: Lack of human validation for outcome grading and conflict management (L248, L382).
We thank the reviewer for this valuable suggestion. Our original paper already reports human validation of two automated judgments: Deliberate Interference (DIR) and Deceptive Concealment. Manual audits of 50 randomly sampled trajectories achieved 98% and 100% human–evaluator agreement, respectively.
To directly address the reviewer’s concern, we conducted an additional human audit of 50 trajectories covering requested-task success, incumbent health, successful preemption, conflict recognition, conflict reporting, and user deferral. Human annotators independently reviewed the complete execution trajectories, including task instructions, tool interactions, and final outcomes, and assigned reference labels according to the evaluation criteria defined in the paper. Disagreements were resolved through discussion. The original and additional validation results are summarized below.
The first two rows report the original audits; the remaining six report the additional audit. These additional results extend human validation to both rule-based outcome grading and LLM-based conflict-management judgments, providing direct evidence of their agreement with human assessment on the audited trajectories.
W2: Ambiguity in the construction of group conflict categories (L190).
Thank you for this question. Our taxonomy was constructed through a two-stage, literature-guided process. First, drawing on established operating-system abstractions [1,2], we compiled an inventory of resource types covering computation, memory, storage, processes, synchronization, and communication, and extended it to analogous daily-life resources. Second, based on resource-management principles and conflict semantics [2–4], we synthesized five conflict mechanisms: Passive Overwrite, Lease/Lock, Name Collision, Elastic Capacity, and Quota Exhaustion.
We then combined these two dimensions—resource types and conflict mechanisms—to construct task instances with different occupancy configurations, resulting in 268 validated cases across 55 resource types and 175 occupancy configurations. We will clarify this taxonomy construction process and its literature foundations in the revised manuscript.