acceptodds
Withdrawn from ICLR 2027

What the Model Sees: A Safety Evaluation of LLM-based Email Assistants in Real Mail Clients

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly used for daily tasks, including email processing and action. Existing evaluations often test phishing detection from fixed inputs, while deployed AI assistants may receive emails only after mail clients render and transform them. We study 21 mail clients, 16 LLMs, four input forms, and six AI browser extensions. We define a run as safe when the assistant recognizes the risk, warns the user, and refuses to act on the message. We separate message accessibility, risk judgment, and user protection, which are often conflated in prior evaluations. Safe rates across 16 LLMs range from 28.6% to 100.0%, while message accessibility across 21 clients ranges from 16.7% to 97.8%. Input representation also matters: the same attack message reaches a safe verdict in 95.0% of screenshot runs versus 54.0% with HTML uploads. Client warnings increase the safe rate from 57.5% to 86.1%, while a lightweight risk instruction increases it from 68.1% to 81.4%. Across six AI browser extensions, only 24.4% of delegated email tasks result in a warning or refusal. These results show that safe email delegation depends on message exposure, input representation, and contextual safety signals. We recommend that mail services and clients provide AI assistants with both risk assessments and readable message content.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, email security, AI safety evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q4HGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the safety of LLM-based email assistants when handling phishing emails and prompt injection attacks in real-world email clients. The authors introduce an Observation-Aware Safety Evaluation perspective, arguing that conventional evaluations typically provide LLMs with fixed email text, overlooking the data processing and delivery pipeline in real-world deployments. To address this gap, the authors construct a benchmark covering 21 email clients, four input formats, and six AI browser extensions. They evaluate 16 LLMs and collect and analyze the experimental results.

Critical strengths

Practical Relevance. The research problem addressed in this paper is practically relevant. The authors consider realistic scenarios and evaluate commonly used software and browser extensions, making their experimental setting more representative of real-world applications than existing benchmarks. Real-world deployments differ substantially from the simplified settings adopted in many prior studies, and this work helps address the lack of attention to these differences.

Critical weaknesses
  1. Limited Novel Insights and Generalizability. The paper's contributions are primarily descriptive, and its main findings (e.g., different input representations affect model safety) are relatively intuitive, lacking deeper analysis of the underlying mechanisms. Furthermore, although the authors emphasize the general significance of Observation-Aware Safety Evaluation, the experiments are restricted to email processing, providing insufficient evidence that the findings generalize to other agent tasks.

  2. Insufficient Scale of the Core Experiments. Although the paper evaluates a large number of models and email clients, the core controlled experiments comparing different input representations rely on only a single attack email, repeatedly tested across different models and prompting conditions. Such a limited number of independent test samples raises concerns about the robustness and representativeness of the findings.

  3. Limited Diversity of Attack Strategies. The authors cite extensive prior research on prompt injection in the Related Work section, yet the experiments primarily rely on straightforward instruction-based injections without sufficiently considering more sophisticated attack strategies. For example, Liu et al. (ACL 2026) investigate Reasoning Hijacking and demonstrate a Criteria Attack in spam detection, where misleading evaluation criteria are injected to manipulate model decisions. Why are such existing attack strategies not considered in this work? The limited diversity of attacks also weakens the conclusions regarding the safety of different input representations.

  4. Lack of False Positive Evaluation on Benign Emails. Emails without prompt injections may still contain phishing or social-engineering content, and the paper does not separately report false positive rates on genuinely benign emails. Consequently, the reported improvements from risk-mention prompts may simply reflect an increased tendency to issue warnings rather than a genuine improvement in risk discrimination.

Questions for authors
  1. Statistical Significance. Could the authors provide confidence intervals or statistical significance tests for the main experimental results to establish whether the observed differences across models, input representations, and experimental conditions are statistically reliable?

  2. Annotation Reliability. Appendix C states that all verdict labels were assigned by the authors without independent annotation by a second annotator. Were any alternative measures taken to validate the reliability of these labels? Could the authors provide independent validation results or an inter-annotator agreement analysis?

Reviewer_kg3GOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the security of LLM-based email assistants in real mail client environments, focusing on how email rendering, content extraction, and input representations affect the assistant's ability to recognize phishing and prompt injection attacks. The paper introduces an end-to-end evaluation framework that distinguishes message coverage, conditional safety, and deployment safety, aiming to separate failures in accessing email content from failures in security judgment.The authors evaluate 21 mail clients, 16 LLMs, and 6 AI browser extensions across different input representations, including screenshots, extracted text, uploaded images, and HTML files. The experiments also examine the effects of client-provided security warnings and risk-aware prompts. The results show that safety judgments vary substantially across input representations and mail clients, and that explicit risk warnings can improve risk recognition. The browser extension evaluation further reveals limitations in end-to-end safety performance.Overall, the paper provides an empirical characterization of how email processing pipelines and interface-level factors influence the security behavior of LLM-based email assistants.

Critical strengths

The paper provides a relatively broad empirical evaluation of LLM-based email assistants across 21 mail clients, 16 LLMs, and 6 AI browser extensions. This is a useful practical contribution, as it moves beyond isolated prompt injection tests and considers how email content is processed and presented in actual client environments. The cross-system evaluation offers a concrete picture of the variability in accessibility and safety judgments across different deployment settings.The distinction between message coverage, conditional safety, and deployment safety is also useful. Rather than reporting only the fraction of safe responses, the paper attempts to separate failures in accessing email content from failures in recognizing security risks. This makes the evaluation more informative for understanding where practical email assistant systems fall short.

Critical weaknesses
  1. Limited novelty and insufficient mechanistic insights. The paper's main findings are largely expected: input representations affect model judgments, incomplete context causes failures, and security warnings improve risk recognition. While the evaluation across real email clients has practical value, the contribution beyond existing research appears limited. The proposed metrics mainly decompose end-to-end success into accessibility and conditional safety, without introducing substantial methodological novelty. More importantly, the paper reports performance differences without explaining which rendering or extraction mechanisms cause them. Although the paper includes non-injected counterparts as controls, these do not necessarily constitute benign-email controls. Without a systematic evaluation of false positives on legitimate emails, it remains unclear whether security warnings improve threat discrimination or merely encourage more conservative responses. Overall, the work remains largely descriptive and provides limited new scientific insight.
  2. Limited validity and generalizability of the empirical evaluation. Several experimental limitations weaken the paper's conclusions. First, the controlled input-form comparison uses only one fixed attack message, making it difficult to generalize the findings to other attacks or email structures. Second, although the paper separates coverage from conditional safety, its deployment safety metric combines message accessibility with security judgments. Consequently, the reported 24.4% rate should not be interpreted as a direct measure of vulnerability to successful attacks, since unreadable messages and unsafe responses represent different failure modes. Third, the browser-extension experiments assess risk recognition, refusal behavior, and stated intentions rather than actual attack execution. Although this limitation is acknowledged, it leaves a substantial gap between the measured outcomes and the paper's broader real-world security claims. Finally, some conditions contain only 10 readable runs, and the lack of uncertainty estimates makes cross-system comparisons less convincing. These limitations substantially weaken the evidence supporting the paper's broader real-world security claims.
Questions for authors

1.Beyond applying existing safety evaluation methods to email clients, what substantive new findings does this paper provide? Can the authors identify the specific rendering or extraction mechanisms responsible for the observed security failures?

2.Section 6.2 acknowledges that the input-form comparison uses only one fixed attack message. How can the authors establish the generalizability of these findings? Did the evaluation include genuinely benign emails to measure false-positive rates under security warnings and risk-aware prompts? Additionally, were confidence intervals or statistical significance tests used to assess the reliability of the reported differences across systems? 3.How do the authors justify interpreting unreadable emails as failures in deployment safety rather than as accessibility failures? Given that the sandbox prevents actual execution of malicious instructions, to what extent can the reported 24.4% deployment safety rate support conclusions about real-world security? Given that the sandbox prevents actual execution of malicious instructions, how do the authors justify interpreting the reported 24.4% deployment safety rate as evidence of real-world security risks?

Reviewer_L5abOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates the safety of LLM-based email assistants across mail clients, models, input representations, and browser extensions -- primarily intervening on the path where an AI assistant directly consumes the email client rendered content. Authors show that the input form, client-generated warning, lightweight risk instructions change levels of security and safety across the board -- with experiments on 16 LLMs, 21 email clients, four input forms and six browser extensions.

Critical strengths
  1. Targeted intervention on specific failure mode and End-to-end evaluation on that: Authors separate out an interesting failure mode/attack mode, and evaluates message accessibility, risk judgment, and deployment safety.
  2. Broad empirical study: Empirical study seems broad, spanning several LLM-client-extension combinations.
  3. There are some interesting findings for that can be useful for AI deployment.
Critical weaknesses
  1. For, most of the findings, a question rises is how is it interesting to the research community and is it a novel finding? I do not see a clear answer for this. -- Input form is interesting dimension, but has been a subject of interest for a long time in the jailbreak/prompt injection community. The "why" would have been interesting, but that is not discussed by the authors. -- There is also not much discussion and connection to the large bodies of related work other than, the static fixes setup statement.
  2. Authors rely on synthetic payloads and a bounded set of test messages.
  3. There is no real action execution by browser extensions. So, the severity of the risks considered are also hard to quantify.
Questions for authors
  1. L192: I am unsure why the observation function is introduced in such a way. The figure and previous writings make it simple enough. I don't see the value of 4.1 as such.
  2. L244: This list reference is broken. This is important for evaluation. The list seems in the main paper.
  3. L288: Why deployment safety is separately reported here. How does it help us interpret the results?
  4. L408: Given such cases of increased risk recognition due to explicit "risk-mention", have you done checks for safe messages as well? If the detected risk false positive percentage is going up, this result should be interpreted differently.
  5. L419: Why is the readability analysis even important? It seems to show that 48% of the combinations are not practically useful to consider. Can you explain how should it be interpreted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.