acceptodds
Under review as a conference paper at ICLR 2027

APEX-Harm: Evaluating Agent Safety in Shared Professional Workspaces

Abstract

Professional agents increasingly work through command-line interfaces and multiple tools over shared workspaces containing files, messages, and business applications. These environments create safety risks that are difficult to capture with attacks placed at a single, known read location. We introduce APEX-Harm, a benchmark for evaluating prompt and script injection against long-horizon agents performing finance, consulting, and legal tasks in large workspaces. Each task contains 115–328 files on average, together with email, calendar, chat, document, spreadsheet, PDF, and code-execution tools exposed through MCP servers. The benchmark includes six attack categories covering static and dynamic injections, script replacement, and user-prompt attacks, and evaluates attack success, exposure, and task performance. We evaluate 12 frontier models on 180 task instances per model. Opus 5 and Sonnet 5 have the lowest overall attack success rates (8% and 20%), while script replacement and prompt-suffix attacks remain substantially more effective, averaging 81-84% and 73% attack success, respectively. Gemini 3.8 Flash achieves the highest mean task score (0.877) while still exhibiting a 33% attack success rate. These results show that completing the requested task is not sufficient evidence of safe behavior: an agent can produce a correct deliverable while also performing an unauthorized action in the workspace.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent safety, prompt injection, script injection, shared workspaces

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iV5qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces APEX-Harm, a benchmark for evaluating long-horizon agent safety in shared professional workspaces. Built on APEX-Agents, it includes six categories of prompt and script injection attacks, 24 attack goals, and 180 attack instances across investment banking, law, and management consulting. Twelve frontier models are evaluated using exposure rate, attack success rate, and task utility. The results show that successful task completion does not imply safe behavior and that script-based attacks remain particularly effective.

Critical strengths
  1. The paper studies an important and realistic setting involving large workspaces, multiple applications,and long-horizon tool use.
  2. Separating exposure, attack success, and task utility provides a useful characterization of agent safety failures.
  3. The benchmark represents substantial engineering effort, integrating diverse attack vectors, dynamic file replacement, application-level interactions, and programmatic safety checks.
Critical weaknesses

1.The 180 attack instances are derived from only 15 base tasks across three domains. The resulting samples are therefore correlated, and the generalizability of the conclusions remains limited. 2. Each model–attack-instance combination is evaluated only once. Given the stochasticity of long-horizon agent trajectories and tool use, the stability of the reported ASRs and model rankings is unclear. 3. Judge validation is limited to 60 cases labeled by a single human annotator.

Questions for authors
  1. How stable are the reported results given that each model–attack pair is evaluated only once?
  2. How well do the findings generalize beyond the 15 base tasks and three professional domains?
  3. Could the authors clarify the reliability of the LLM judge given the limited single-annotator validation?
Reviewer_XgKpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

APEX-Harm evaluates agent safety in shared professional workspaces across investment banking, law, and consulting. It introduces 180 attack instances from 15 tasks, which covers 6 categories of prompt and script injection. Result shows that strong task performance can coexist with unauthorized actions. This paper shows that vulnerabilities involving attacker-controlled scripts and internal information disclosure, demonstrating that task completion alone is insufficient for assessing agent safety.

Critical strengths
  1. realistic environments and explicit threat models. The paper proposes large workspaces, long-horizon tasks, and multiple application, which relate to attacker knowledge, permissions, and user authorization. This enables evaluation of security boundaries that simpler injection settings can overlook.

  2. informative evaluation beyond attack success. Measuring exposure, attack success, and task utility separately helps distinguish attacks that agents never encounter from those they actively resist.

Critical weaknesses
  1. limited tasks and corresponding analysis. There are only 15 tasks in total, covering 3 domains. Although authors design 12 cases for each task, the limited source tasks will introduce substantial variability and bias into the evaluation results. The paper fails to compare the results across the proposed 3 domains. Also, the paper claims that each domain contains 3 easy, 1 medium, 1 hard tasks, while failing to do corresponding analysis on different difficulty levels. This point weakens the paper's claims.

  2. Difference from existing safety benchmarks. The paper designs 5 attacker ability dimensions. But once the malicious content has been injected into the workspace and the test case finalized, the remaining evaluation procedure appears to differ little from existing agent safety evaluations. Moreover, the proposed dynamic attacks largely rely on a single strategy, namely simulating file corruption, which substantially limits their scope. I would like the authors to clarify which additional aspects of model safety this benchmark assesses beyond those covered by existing benchmarks. This is a concern of novelty.

Questions for authors
  1. How are the error bars in Figure 4 calculated?
  2. Are the results in Table 21 a subgroup breakdown of those in Table 3?
Reviewer_mZ1DOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a benchmark called APEX-Harm: it tests whether malicious prompts/scripts can steer agents into unauthorized actions. Tasks covered: finance, consulting and legal across large workspaces.

  • Models/tasks: 12 frontier models, 180 task instances each, 115-328 files per task
  • Results: attack success rates: Opus 5=8% and Sonnet 5=20%, Gemini 3.8 Flash=33% Key takeaway: a correct deliverable of the task does not prove safe/secure execution.
Critical strengths
  • the paper addresses an important security problem: agents performing legitimate professional tasks can be manipulated into unauthorized actions.
  • tried broad attack coverage within realistic workspaces
  • evaluated a variety of models: open-weight and proprietary
  • the appendix gives substantial detail about attacks and grading.
Critical weaknesses

There are no critical weaknesses, suggestions: would love to see analyses on:

  • over-refusal: task performance under attack is useful but matched benign scripts would be interesting to see
  • dynamic attacks: runtime file modification is different from an adversary iteratively optimizing attacks against a model.
Questions for authors

what false-positive checks were used for attachment and internal-disclosure rules?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.