- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates the safety of LLM-based email assistants when handling phishing emails and prompt injection attacks in real-world email clients. The authors introduce an Observation-Aware Safety Evaluation perspective, arguing that conventional evaluations typically provide LLMs with fixed email text, overlooking the data processing and delivery pipeline in real-world deployments. To address this gap, the authors construct a benchmark covering 21 email clients, four input formats, and six AI browser extensions. They evaluate 16 LLMs and collect and analyze the experimental results.
Critical strengths
Practical Relevance. The research problem addressed in this paper is practically relevant. The authors consider realistic scenarios and evaluate commonly used software and browser extensions, making their experimental setting more representative of real-world applications than existing benchmarks. Real-world deployments differ substantially from the simplified settings adopted in many prior studies, and this work helps address the lack of attention to these differences.
Critical weaknesses
-
Limited Novel Insights and Generalizability. The paper's contributions are primarily descriptive, and its main findings (e.g., different input representations affect model safety) are relatively intuitive, lacking deeper analysis of the underlying mechanisms. Furthermore, although the authors emphasize the general significance of Observation-Aware Safety Evaluation, the experiments are restricted to email processing, providing insufficient evidence that the findings generalize to other agent tasks.
-
Insufficient Scale of the Core Experiments. Although the paper evaluates a large number of models and email clients, the core controlled experiments comparing different input representations rely on only a single attack email, repeatedly tested across different models and prompting conditions. Such a limited number of independent test samples raises concerns about the robustness and representativeness of the findings.
-
Limited Diversity of Attack Strategies. The authors cite extensive prior research on prompt injection in the Related Work section, yet the experiments primarily rely on straightforward instruction-based injections without sufficiently considering more sophisticated attack strategies. For example, Liu et al. (ACL 2026) investigate Reasoning Hijacking and demonstrate a Criteria Attack in spam detection, where misleading evaluation criteria are injected to manipulate model decisions. Why are such existing attack strategies not considered in this work? The limited diversity of attacks also weakens the conclusions regarding the safety of different input representations.
-
Lack of False Positive Evaluation on Benign Emails. Emails without prompt injections may still contain phishing or social-engineering content, and the paper does not separately report false positive rates on genuinely benign emails. Consequently, the reported improvements from risk-mention prompts may simply reflect an increased tendency to issue warnings rather than a genuine improvement in risk discrimination.
Questions for authors
-
Statistical Significance. Could the authors provide confidence intervals or statistical significance tests for the main experimental results to establish whether the observed differences across models, input representations, and experimental conditions are statistically reliable?
-
Annotation Reliability. Appendix C states that all verdict labels were assigned by the authors without independent annotation by a second annotator. Were any alternative measures taken to validate the reliability of these labels? Could the authors provide independent validation results or an inter-annotator agreement analysis?