- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
ACE systematically investigates whether LLM agent security detection should extend from application-level intent to OS-level execution, using paired data and cross-model experiments to characterize the discriminative power and complementarity of different evidence layers.
Critical strengths
The paper addresses an important and underexplored problem in agent security. Its paired-evidence benchmark and systematic evaluation across attack mechanics and detector families provide valuable empirical insights into the complementary roles of application- and kernel-level evidence.
Critical weaknesses
Although ACE covers diverse attack mechanisms, its evaluation primarily relies on hand-crafted attacks in controlled environments. Further validation in real-world deployments could strengthen the generalizability of its findings. Compared with prior execution-level security studies such as AgentSight, the contribution lies mainly in systematic empirical characterization, and its distinct value could be more clearly emphasized.
Questions for authors
Could the authors clarify how the findings generalize from hand-crafted attacks to naturally occurring attacks?
3 comments
Response to Reviewer aw4X for [C2]
[C2] "Compared with prior execution-level security studies such as AgentSight, the contribution lies mainly in systematic empirical characterization, and its distinct value could be more clearly emphasized."
[Response]
We agree this should be better clarified and address this overlapping concern in our response to Reviewer TnAv’s Comment 1. Please refer to that response here.
Thank you for the detailed clarification. I appreciate the authors pointing out the specific empirical evidence for generalization. I also understand the practical difficulties of evaluating naturally occurring incidents. I am satisfied with this explanation.
Response to Reviewer aw4X for [C1, Q1]
[Response]
We agree that evaluation in real-world deployments would be valuable. In ACE, we use controlled attacks executed through real agent and tool runtimes to compare application and kernel evidence across attack mechanics. Section 2.5 describes 31 real third-party MCP servers alongside built-in tools, spanning six delivery-vector families. This gives us paired observations from the same sessions, explicit execution checks, and benign workloads that exercise the same tool surfaces (Appendices E.5 and E.7), which support consistent comparisons of the evidence layers.
We test generalization in two ways. First, the seven-fold evaluation excludes each tested attack family from training and LLM demonstrations (Section 3.3). Cross-View exceeds App-View for all ten detectors, with every pointwise paired 95% interval above zero (Appendix H.4, Table 12). Second, ACE-XA replaces both the agent model and execution scaffold while conditioning detectors only on ACE. Three of four tested detector-family representatives retain mean AUROC above 0.75 (Appendix M, Table 27). These results support the usefulness of execution evidence across held-out attack families and the tested alternate runtime.
For naturally occurring incidents, a useful evaluation needs paired execution records, or a faithful reconstruction of the affected environment, together with comparable benign activity. Replaying an incident in a different harness can change tool behavior, filesystem access, and network interactions, which are the signals we measure. We would therefore need to verify that the reconstruction preserves the relevant execution behavior as well as the malicious objective.
Case studies of disclosed incidents would provide a useful complement. However, assembling a sufficiently diverse collection is challenging: each incident must be documented in enough detail for faithful reconstruction, and the collection must cover distinct attack mechanics to support broader conclusions. Selecting only the incidents that are readily available and reproducible could disproportionately represent certain attack types. Such case studies would establish performance on those specific incidents, while assessing generalization would require a dedicated effort to collect, reconstruct, and validate a broader set of attacks.
Our focus in this study is controlled coverage and explicit transfer tests. Evaluation on naturally occurring incidents, using paired telemetry or reconstructions with validated behavioral fidelity and comparable benign workloads, remains a valuable direction for future work. We will clarify the scope of our generalization evidence and this future direction in the final paper.