- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces APEX-Harm, a benchmark for evaluating long-horizon agent safety in shared professional workspaces. Built on APEX-Agents, it includes six categories of prompt and script injection attacks, 24 attack goals, and 180 attack instances across investment banking, law, and management consulting. Twelve frontier models are evaluated using exposure rate, attack success rate, and task utility. The results show that successful task completion does not imply safe behavior and that script-based attacks remain particularly effective.
Critical strengths
- The paper studies an important and realistic setting involving large workspaces, multiple applications,and long-horizon tool use.
- Separating exposure, attack success, and task utility provides a useful characterization of agent safety failures.
- The benchmark represents substantial engineering effort, integrating diverse attack vectors, dynamic file replacement, application-level interactions, and programmatic safety checks.
Critical weaknesses
1.The 180 attack instances are derived from only 15 base tasks across three domains. The resulting samples are therefore correlated, and the generalizability of the conclusions remains limited. 2. Each model–attack-instance combination is evaluated only once. Given the stochasticity of long-horizon agent trajectories and tool use, the stability of the reported ASRs and model rankings is unclear. 3. Judge validation is limited to 60 cases labeled by a single human annotator.
Questions for authors
- How stable are the reported results given that each model–attack pair is evaluated only once?
- How well do the findings generalize beyond the 15 base tasks and three professional domains?
- Could the authors clarify the reliability of the LLM judge given the limited single-annotator validation?