acceptodds
Under review as a conference paper at ICLR 2027

AgentPhishing: A Benchmark of LLM Email Agents Under Adaptive Attack

Abstract

Computer-use agents increasingly handle private information and act on users’ behalf across desktop and web applications, including email. Phishing therefore poses privacy and security risks that depend not only on whether agents recognize malicious messages, but also on how they respond. Existing phishing datasets were developed primarily for conventional detection, leaving agent susceptibility to persuasive malicious requests insufficiently characterized. To expose vulnerabilities that pre-existing datasets may miss, we introduce AgentPhishing, a benchmark for evaluating whether persuasive phishing attacks can evade detection and induce LLM email agents to comply with malicious requests. We develop PhishOpt, which optimizes reusable email-generation skills against individual agents under fixed benchmark constraints and a realism constraint. We analyze the generated emails using Cialdini’s persuasion principles to identify strategies associated with agent failures and use these findings to construct a universal generation skill. Our evaluation shows that agents can handle pre-existing phishing emails safely yet remain susceptible to attacks adapted to agent behavior, while in an AgentPhishing evaluation of nine agents, four exhibit phishing-detection rates below 10% and malicious-request compliance rates above 45%. These results highlight the limits of inferring agent safety from conventional phishing evaluations and the need to assess how agents respond to persuasive malicious requests.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Agents, Safety, Privacy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W415OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces “AgentPhishing”, which is a benchmark for evaluating phishing detection effectiveness and how often agents comply with malicious requests under social-engineering attacks in email tasks. The authors also present “PhishOpt”, an attack method that adapts existing work “SkillOpt” to optimize reusable prompts for generating emails in a pipeline. The pipeline also uses Cialdini’s persuasion principles within the pipeline to label generated emails and use those to write a universal generation skill/prompt. The authors present experiments across six models used to develop the attacks and nine additional models used for evaluation. The work shows that some models handle more basic phishing attacks safely but fail on more tailored attacks.

Critical strengths

The paper addresses an important practical problem and usefully separates phishing detection from compliance with malicious requests. The results show why conventional phishing evaluations may give an incomplete picture of agent safety.

Critical weaknesses

The contribution over existing work seems narrow. PhishOpt adapts SkillOpt, while AgentBait already studies social engineering against agents and PEEK uses persuasion analysis for phishing generation. I’m not convinced that this work makes a substantial contribution. The paper focuses on phishing generation by combining existing methods and adapting the pipeline narrowly.

The evaluation lacks validation. Email realism and persuasion labels are assessed by LLMs without reported human validation. This leaves uncertainty about both the plausibility of the attacks and the reliability of the persuasion analysis because evaluating LLMs with imperfect LLMs leaves room for unreliable evaluation.

Existing adaptive methods are not used for comparison. The main baselines are static emails and an unoptimized prompt. These comparisons do not show that PhishOpt offers a meaningful improvement over other optimization approaches such as TextGrad.

Questions for authors
  1. What is novel in this work beyond combining AgentBait, SkillOpt, and PEEK and deriving a prompt based on including Cialdini’s persuasion principles?
  2. How does PhishOpt compare with an existing adaptive attack method or a generic prompt optimizer under the same generation and target-query budgets?
  3. What evidence validates the LLM realism judgments and Cialdini labels? I think the evaluation should be stronger than relying on LLMs that might themselves be imperfect.
3 comments

W1: We thank the reviewer for asking us to clarify our contribution. Prior work like PEEK evolves phishing emails against a classifier to train phishing detectors, and its code is not publicly available. AgentBait studies social engineering against agents, but its attacks are predefined and fixed before the agents are observed. PhishOpt is the first method that adapts attack generation to the evaluated agent, using an objective score combining detection, compliance, and F1 as feedback.

To address concerns between PhishOpt and similar Phishing detection methods, we compared directly with AgentBait predefined attacks. We converted 73 released email-scenario attacks to text and evaluated them against seven agents. From the table, AgentBait's attacks are about as detectable as our base skill, while PhishOpt evades detection on six of the seven agents.

AgentAgentBait (73)Our base (12)PhishOpt (12)
Qwen 3.7 Plus0.861.000.00
Grok 4.200.960.920.42
GPT-5.40.740.920.58
Gemini 3.5 Flash0.841.000.58
Sonnet 51.001.001.00
gpt-5-mini (held out)0.730.500.00
qwen-max (held out)0.380.170.00
Mean0.790.790.37

PhishOpt = each agent's optimized skill (original 5), universal skill (held-out 2).

W2(Regarding human validation on realism and persuasion labels): We agree that LLM judgement without human validation leaves uncertaintity. To validate both judges, one human annotator hand labbeled a sample of 30 emails for realism and persuaion principles, using the same criteria used by the LLMS (table below). For persuasion principles, LLM and human agreed on 89% of the time with the strongest agreement on commitment and consistency. For realism, both rate generated emails as realistic and both degraded on emails below the 0.7 gate.

PrincipleAgreementCohen's κPresent: LLM / human
Authority100%undefined (present in every email for both)100% / 100%
Reciprocity96%0.8414% / 11%
Commitment & consistency89%0.7379% / 68%
Social proof89%0.5114% / 11%
Scarcity / urgency82%0.5075% / 79%
Liking75%0.5054% / 57%
Mean89%0.62 (5 defined principles)

Realism, LLM judge vs human (30 items; human 1–5 mapped to 0–1):

LLM judgeHuman
Generated emails, mean0.900.81
Garbled controls, mean0.330.28
Controls below the 0.7 gate2/22/2
Generated emails below the gate1/285/28

W3: We thank the reviewer for this suggestion. To address this concern, we ran TextGrad on GPT-5.4 with the same base skill, generator, engine, model, feedback, budget in two configurations. One where we use standard textGrad and the other where we use TextGrad inside our pipeline where we replace phishOpt editor with TextGrad Update step and keep everything else the same. We evaluated each final skill on three fresh batches (table below).

SkillTest J (3 batches)Mean ± SDDetectionFalse-positive rateCompliancePhishing realism (mean; below 0.7 gate)
Base0.022 / 0.085 / 0.0790.062 ± 0.0340.940.0800.77; 11/36
TextGrad, no gate0.045 / 0.145 / 0.1450.112 ± 0.0580.890.0800.80; 4/36
TextGrad, in PhishOpt loop0.085 / 0.187 / 0.2380.170 ± 0.0780.790.4200.89; 1/34
PhishOpt0.253 / 0.233 / 0.2420.243 ± 0.0100.660.000.030.87; 3/35

From the table, PhishOpt scored highest on every test batch and substantially reduced detection rate without raising false positives. TextGrad in the PhishOpt pipeline increased in score but it mainly came from GPT-5.4 flagging more benign emails which an increase in false positive. This comparison covers one target and one run per method, and TextGrad used more engine calls than PhishOpt (three per update vs. one); we will extend it to additional targets in the revision.

Reviewer_W415OpenReview ↗

Thank you for your response and the extra experiments.

W1 (novelty): This remains my biggest concern. I still feel that the contribution is very narrow. PhishOpt is built on SkillOpt. Using a target’s feedback to improve an attack is also not new, as your own related-work section shows (AutoInject, MUZZLE, SpearBot). So the contribution still looks like existing techniques applied to a new setting, where considerable work already exists.

W2: I appreciate the added human validation. However, the human check relies on a single annotator, so there is nobody to compare their labels against. I think two or more independent annotators are needed for the check to be convincing. I also noticed that the annotator rated 5 of the 28 emails below your realism threshold, compared with 1 for the LLM judge, so the judge may be more lenient than a human.

W3: Thank you for adding this baseline. As you note, the TextGrad comparison covers one agent and one run per method. The gap between PhishOpt and TextGrad inside your pipeline seems small compared with the variation between batches. That is too little to conclude that PhishOpt is better. Results on more agents, with several runs per method, would make the comparison convincing. It may also be helpful to report results with unmodified SkillOpt.

I am keeping my rating.

W1(Novelty): Our contribution is not the optimization mechanism but the threat model it measures and the findings it produces: what a tool-using email agent does with a persuasive but non-injecting message inside a task loop. To our knowledge, no prior work measures this quantity. PhishOpt introduces a phishing-specific objective (detection and compliance), a realism gate that enforces persuasion-only generation, locked synthetic-only generation constraints, and a universal skill built from persuasion analysis.

MethodWhat it optimizesFeedback signalTargetChannel
PEEK (Chen et al., 2024)phishing generatorreal-vs-synthetic discriminatorphishing classifierpersuasive email
SpearBot (Qi et al., 2025)individual emailsdetector critiquesphishing detectorpersuasive email
AutoDAN-Turbo (Liu et al., 2025)jailbreak strategy librarytarget responsesLLM (jailbreak)jailbreak prompt
RL-Hammer (Wen et al., 2025)attacker-model weights (RL)black-box target rewardLLM agentprompt injection
AutoInject (Chen et al., 2026)injection suffixes (RL)target rewardLLM agentprompt injection
MUZZLE (Syros et al., 2026)injection location + payloadexecution trajectoryweb agentprompt injection
PhishOpt (ours)natural-language skillagent detection + complianceemail agent in a task looppersuasion only (realism-gated)

W2: We added 2 more annotators scored against the LLM judges. With realism, all three rated the generated emails as realistic with means around 0.81-0.91 and the judge placed both degraded controls below 0.7. With persuasion, one annotator agreed strongly with the key (mean κ = 0.62), a second moderately (κ = 0.30), and a third weakly (κ = 0.11). Agreement was consistent on scarcity/urgency (κ = 0.48–0.62 across all three) and dropped on the remaining principles for the two weaker annotators. This bounds the precision of the fine-grained persuasion coding but not the behavioral findings, which are computed from agent actions rather than these labels.

RealismAnswer key (LLM)Annotator AAnnotator BAnnotator C
Generated emails, mean0.900.910.810.82
Degraded controls, mean0.330.810.280.84
Controls below the 0.7 gate2/21/22/20/2
Generated emails below the gate1/282/285/281/28
PrincipleKey presentAnnotator AAnnotator BAnnotator C
Authority1.00—*—*—*
Scarcity / Urgency0.750.480.500.62
Commitment & Consistency0.790.480.730.00
Social proof0.140.180.51−0.24
Liking0.540.110.500.14
Reciprocity0.140.280.840.13
Mean κ0.300.620.11

* Authority: the key marks it present on all 28 emails, so κ is undefined (no variance); raw agreement is 82–100%.

W3: We have extended the experiment to five agents and split it into two. Experiment 1: is PhishOpt better than a generic adaptive optimizer? Experiment 2: does the advantage come from the editor or the loop?

Experiment 1: We hold the base skill, generator, engine model, feedback, and budge fixed. We ran both methods against five target agents. From the table below, we can see that PhishOpt exceeds standard TextGrad on every agent suceptile attacks on Qwen, GPT, Grok. There are two exceptions with Gemini and Sonnet, where they are robust to all methods.

Target agentBaseTextGradPhishOpt
qwen3.7-plus0.0900.1480.519
gpt-5.40.0620.1120.243
grok-4.200.0900.1390.252
gemini-3.5-flash0.0690.0740.024
claude-sonnet-50.0350.0430.037

Experiment 2: To determine whether PhishOpt's advantage comes from its editor or from the surrounding loop (the realism gate and best-so-far selection), we held the loop fixed and replaced only the editor with TextGrad's. When TextGrad is given PhishOpt's loop, its performance rises sharply, showing that the loop is a core strength of our approach. PhishOpt's editor performed well on Qwen and GPT, while on Grok, TextGrad-in-loop scored higher. This does not reflect a more effective attack: the gain came from the agent incorrectly flagging legitimate emails as phishing without causing it to act on a genuine phishing request. An independent judge also rated the realism of every email, confirming that no method improved its score by reducing realism.

Target agentTextGrad-in-loopPhishOpt
qwen3.7-plus0.1500.519
gpt-5.40.1700.243
grok-4.200.3130.252
gemini-3.5-flash0.1080.024
claude-sonnet-50.0440.037
Reviewer_YQGiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a benchmark for evaluating how AI email assistants respond to phishing emails. Its attack-generation framework, PhishOpt, repeatedly improves email-writing instructions using feedback from a target assistant. The evaluation compares existing phishing emails, unoptimized generated emails, and optimized attacks. It also tests whether the generated attacks work against models excluded from optimization. The results show that assistants can handle existing phishing examples safely while remaining susceptible to tailored requests. The paper further analyzes persuasion tactics and uses those findings to construct a reusable generation prompt.

Critical strengths

The benchmark reflects a realistic adversarial setting. An attacker may have access to the target assistant through an open-weight deployment or public API and use its responses to refine phishing attempts. Optimizing attacks against target feedback therefore captures risks that evaluations using fixed, pre-existing emails may miss.

Critical weaknesses

The distinction between optimization results and held-out evaluation could be clearer. Appendix A notes that own-target cells can reuse retained best results, which may overstate the performance expected on newly generated emails.

Questions for authors

Could the authors clarify whether the optimized results in Figure 3 reuse scores obtained during skill selection or come from held-out evaluation after selection? If they reuse selection scores, do fresh-batch evaluations confirm the reported improvements?

2 comments

Thank you for the update. I will keep my score after the clarification.

Reviewer_ViFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AgentPhishing introduces a benchmark to evaluate phishing detection and compliance under adaptive attacks. PhishOpt optimizes attacks against six models, including Qwen3.7-Plus, Kimi K2.5, Grok 4.20, GPT-5.4, Claude Sonnet 5, and Gemini 3.5 Flash. It also tests transfer to nine held-out models, including GPT-5.6, Claude Opus 4.8, and Gemini 3.1 Pro. Four models show detection below 10% and compliance above 45%.

I find the distinction between detection and compliance important, and adaptive testing is a useful contribution over static phishing benchmarks. However, my main concern is with what is actually measured. Compliance is based on selected actions and drafted responses rather than actual tool execution. Similarly, selecting safe actions does not necessarily mean the agent recognized phishing. This raises questions about whether the results reflect actual security failures or just intended behavior. While the framework is useful, stronger validation of tool execution and resulting state changes is needed. Hence, I lean toward a weak rejection.

Critical strengths

The benchmark distinguishes between phishing detection and compliance with malicious requests, and demonstrates that static datasets may miss vulnerabilities exposed through adaptive attacks.

Critical weaknesses

Compliance is measured based on what the agent selects or writes, rather than actually executing the action. Saying it would send confidential information is different from actually sending it. So it is difficult to establish real security failures.

The benchmark assumes actions like archive, ignore or unsubscribe means the agent detected phishing. But these actions may happen for other reasons, or the agent may detect phishing and respond differently. This raises questions on accuracy of detection scores.

The optimization combines detection failure and classification scores which partly measure the same behavior. It is not clear whether the results reflect actual agent vulnerabilities or how the benchmark scores responses.

Questions for authors

How did the authors validate that the agent actually detected phishing? For example, choosing to archive, ignore or unsubscribe an email does not necessarily mean it recognized phishing. Similarly, an agent may detect phishing but take some other safe action. How would these differences affect the reported detection scores?

Can the authors test the benchmark in an environment where agents actually execute tool calls?

1 comment

W1: We thank the reviewer for addressing the distinction between response-level compliance and executed actions. In Section 4, it explicitly states that our interface evaluates whether an email selects a malicious action or generates a compliant reply, and in Section 6 it states that measured compliance does not establish completed real-world harm. Our previous results show what an agent decides and if it writes when it receives a phishing email. They do not show that credentials were actually sent or that an account was compromised.

To address this concern, we added a sandbox experiment where agents are able to execute actions; more specifically, the agent is able to read credentials and files and send messages. All passwords/API keys are unique canary tokens, and we count a leak if a canary appears in a sent message or in a shared location (ticket, doc, etc.). The agents get a standard inbox assistant prompt with no additional safety guidance.

Through the 167 runs across 7 agents, with 12 emails per agent with baseline against attack skill, we saw that agents do execute leaks and not just comply under text. Under the attack skill, four of the seven agents leaked canaries, while Gemini 3.5 Flash, Sonnet 5, and GPT-5.4 did not. 37 of the 167 runs leaked, and response-level compliance did not predict executed leaks exactly for some agents and understated them for others.

AgentBaseOptimized / universalResponse-level compliance (attack skill)
Qwen 3.7 Plus0/129/1211/12
Grok 4.200/127/121/12
qwen-max (held out)5/1212/12 (universal)11/12
gpt-5-mini (held out)0/124/12 (universal)9/12
Gemini 3.5 Flash0/120/123/12
Sonnet 5, GPT-5.40/120/12 (Sonnet 0/11; its batch has 11 phishing emails)0

Pooled per-email comparison:

LeakedDidn't leak
Response-level compliance2621
No response-level compliance11109

W2 (Regarding Action determining phishing): We agree that the passive actions do not, by themselves, tell us that an agent may recognize phishing yet respond differently. To address this concern, we first reran seven agents with a phishing/benign classification field and then we compared each agent's answer with the action it took on the 167 phishing emails.

ActionClassified phishingClassified benign
report1200
reply519
provide_info017
follow_up06
archive / ignore00

From the table, the 5 cases in which an agent answered phishing but still replied are still considered a miss in our metrics. In the 42 where the attacks worked and the agents classified benign and never reported the email. Instead, it either replied, provided information, or deferred it for a follow-up. This shows a recognition failure, not a case of recognizing phishing and responding differently. Overall, the action label and the agent's answer agree on 97% of phishing emails.

W3 ( F1 overlap concerns): We include the F1 term since detection and compliance are computed from the phishing emails alone, while F1 also counts the benign emails, so that the objective reflects whether the agent can tell phishing from benign rather than only disengaging. The F1 term does overlap with detection, and this is ** addressed by the ablation experiment in Section 5.4** where we remove the F1 term, rebalance the objective score, and rescore the existing results. The score gains for Qwen, Kimi, Grok, and GPT remain (0.85, 0.55, 0.51, 0.38), so the increases are not attributable to F1 alone. We will make the overlap and this ablation more explicit in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.