acceptodds
Under review as a conference paper at ICLR 2027

CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents

Abstract

Language model agents are increasingly deployed to automate everyday digital chores from managing emails and social media to handling banking and bills allowing users to step away from supervision. However, this capability also exposes sensitive information to phishing. Safe execution requires distinguishing malicious requests from genuine ones without simply refusing to act. Despite its practical importance, this problem remains underexplored, and it is unclear whether current agents, or existing defenses can achieve it. To study this problem, we first propose CredLeak-Bench, a comprehensive benchmark designed to evaluate how effectively and securely agents automate human workflows when confronted with phishing and identity verification. The benchmark covers both user-directed authentication and autonomous inbox monitoring, where agents are not explicitly instructed to log in. It systematically varies deceptive cues and pairs phishing scenarios with legitimate counterparts, enabling joint evaluation of information leakage and utility on genuine tasks. Within a sandboxed environment, leakage is measured through actual submissions of synthetic information rather than agents’ self-reported behavior. Our evaluation reveals that all tested models including GPT, Claude, Gemini, Qwen are vulnerable to leakage. Agents also disclose sensitive information during autonomous inbox monitoring, demonstrating that phishing can induce disclosure without a user request to authenticate. Furthermore, most evaluated mitigations that reduce leakage also impair performance on genuine tasks, exposing a security–utility trade-off in existing defenses. These findings show why reducing leakage alone is insufficient: effective defenses must prevent unauthorized disclosure while preserving legitimate task completion. CredLeak-Bench provides a controlled framework for measuring both objectives and evaluating progress toward secure, useful agents.

Primary area:
datasets and benchmarks
Keywords:
LLM, LLM agents, Agent safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_59FaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CredLeak-Bench, a sandboxed benchmark that measures credential leakage in LLM agents through actual form submissions rather than model self-reports. Its core contribution is the joint measurement of leak rate and false-refusal rate via paired phishing/legitimate scenarios across directed and autonomous settings, plus a recovery setting evaluates whether an agent can bypass a phishing link and complete the genuine task through a trusted path.

Critical strengths
  1. The measurement is reliable. Each phishing scenario has a legitimate counterpart that differs only in origin. Leakage is verified by matching submitted values against vault credentials in captured network logs, providing more reliable evidence than model self-reports.

  2. A useful recovery setting. As mentioned by the author, avoiding leakage is not the same as completing a task safely. Table 13 shows that Qwen3.5-9B can leak credentials even after reaching a trusted endpoint. This distinction matters because reaching the correct destination does not undo earlier leakage or ensure that subsequent actions are safe.

Critical weaknesses
  1. The paper needs clearer reporting to support its comparisons. For example, the autonomous setting includes only 16 phishing cases per deception cue (Appendix B) and 32 legitimate cases per framing. The Reproducibility Statement says that aggregate rates are stable across reruns, but no results are provided to support this claim. The number of runs per scenario is also unclear. For example, Table 6 reports an autonomous cautious FRR of 99.3% for Sonnet 5. This value would not be possible if each of the 32 legitimate scenarios was run once and FRR was calculated directly over these scenarios. The authors should clarify the number of runs and the denominator used.

There are also several unclear points in the reported metrics. LR and FRR are calculated over “competent” runs, but the paper does not clearly define competence or report how many runs are excluded. Footnote 1 states that the 96 tls_warning cases have no trusted recovery route, yet they are included in the 768-case recovery denominator. This limits the maximum recovery rate to 87.5%. The aggregation in Table 3 also needs explanation. In the directed setting, its values differ from the unweighted means of the values in Tables 6 and 7: Sonnet 5 has LR of 31.5% versus 30.8% and FRR of 26.6% versus 25.2%; Gemini 3.6 Flash has LR of 53.1% versus 53.8%. The authors should explain how these results are calculated and provide evidence of their variation across runs.

  1. The main results should separate settings with and without trusted domain information. Under none and generic_safety, agents are not given the official domain. This makes it difficult to judge whether a domain is authentic, especially for fictional brands. Section 5.5 shows that providing the official domain substantially reduces leakage. For GPT-5-mini in the directed setting, LR drops from 73.5% under none to 23.4% under domain_verify. It is therefore unclear how much of the leakage reflects a model vulnerability and how much comes from missing information needed to identify phishing.

Some cues, such as no_https, raw_ip, and tls_warning, can be detected without knowing the official domain. Section 5.5 reports that leakage on no_https persists under domain_verify. However, the domain_verify prompt in Appendix D.2 only asks agents to compare domains, which match exactly in this case. The threat model in Section 3.1 also requires valid HTTPS, but the prompt does not state this requirement. This makes it difficult to tell whether the agent fails to recognize insecure transport or simply follows an incomplete instruction. The authors should separate cues that require a trusted domain reference from those that do not, report the main results separately for each scaffolding condition, and test whether explicitly requiring valid HTTPS changes the no_https result.

Questions for authors
  1. How do the authors define a “competent” run, and how many runs were excluded for each model and setting?
  2. Could the authors report the headline LR and FRR separately for conditions with and without the official domain information?
  3. In the recovery setting, the headline recovery rate groups escalation to the user with other non-completion outcomes. For example, Opus 4.8 escalates in 722 of 768 cases (Table 13) and is described as never recovering. Why is escalation not credited as a safe outcome, and how would the headline results change if it were?
6 comments
AuthorsOpenReview ↗

Initial Response Pt. 1

We thank the reviewer for their comprehensive review and for acknowledging the usefulness of our proposed recovery setting.

LR and FRR are calculated over “competent” runs, but the paper does not clearly define competence or report how many runs are excluded.

How do the authors define a “competent” run, and how many runs were excluded for each model and setting?

“Competent” refers to a valid evaluation attempt. Exclusions cover cases like API failures, empty or broken tool calls, provider-level declines before submission, and turn-limit exits without a submission. A run is never left out because of what the agent decided, so leaking or refusing has no effect on whether a run counts. Technical failures are retried, and each scenario contributes three valid runs to the final analysis.

The paper needs clearer reporting to support its comparisons. For example, the autonomous setting includes only 16 phishing cases per deception cue (Appendix B) and 32 legitimate cases per framing. The Reproducibility Statement says that aggregate rates are stable across reruns, but no results are provided to support this claim. The number of runs per scenario is also unclear. For example, Table 6 reports an autonomous cautious FRR of 99.3% for Sonnet 5. This value would not be possible if each of the 32 legitimate scenarios was run once and FRR was calculated directly over these scenarios. The authors should clarify the number of runs and the denominator used.

The aggregation in Table 3 also needs explanation. In the directed setting, its values differ from the unweighted means of the values in Tables 6 and 7: Sonnet 5 has LR of 31.5% versus 30.8% and FRR of 26.6% versus 25.2%; Gemini 3.6 Flash has LR of 53.1% versus 53.8%. The authors should explain how these results are calculated and provide evidence of their variation across runs.

We formally re-evaluated the full directed and autonomous suites over three independent repetitions. Table 3 now reports mean ± standard deviation for LR, FRR, Balanced Accuracy, and MCC. Tables 6 and 7 report pooled LR and FRR by framing and scaffolding condition. Each scenario contributes one valid run per repetition. Per repetition, the directed setting contains 256 phishing and 256 legitimate runs per model, and autonomous monitoring contains 128 of each. Across three repetitions, each framing or scaffolding row in Tables 6 and 7 contains 192 runs per class in the directed setting and 96 in autonomous monitoring. Each autonomous deception cue contains 48 phishing runs across repetitions. The standard deviations of LR and FRR in Table 3 range from 0.0 to 1.8 percentage points.

The updated results preserve the main qualitative findings: all seven models disclose information in both settings; cautious framing generally reduces leakage at a cost to legitimate task completion; verification support improves both metrics for most models in directed authentication; and the effects of the autonomous vault policy vary across models. The broad security–utility pattern persists: every model has lower leakage but higher false refusal in autonomous monitoring than in directed authentication.

Table 3. Leakage, false refusal, balanced accuracy, and MCC across models and settings. Values are mean ± standard deviation across three independent repetitions, with metrics computed separately per repetition. LR and FRR are percentages. Each repetition contains 256 phishing and 256 legitimate runs in directed authentication, and 128 of each in autonomous monitoring.

SettingModelLeak Rate %False-Refusal %Balanced Acc.MCC
DirectedOpus 4.831.2 ± 0.06.6 ± 0.00.811 ± 0.0000.641 ± 0.000
DirectedGPT-5-mini45.8 ± 1.112.5 ± 0.00.708 ± 0.0060.442 ± 0.010
DirectedGPT-5.6-luna49.9 ± 0.69.4 ± 0.00.704 ± 0.0030.446 ± 0.005
DirectedSonnet 519.9 ± 0.013.7 ± 0.40.832 ± 0.0020.665 ± 0.004
DirectedGemini 3.6 Flash40.6 ± 0.00.1 ± 0.20.796 ± 0.0010.648 ± 0.003
DirectedQwen3.5-9B66.5 ± 0.52.7 ± 0.00.654 ± 0.0020.399 ± 0.004
DirectedLlama-3.1-8B76.7 ± 0.212.8 ± 0.20.553 ± 0.0020.137 ± 0.005
AutonomousOpus 4.81.8 ± 0.588.3 ± 0.00.549 ± 0.0020.197 ± 0.012
AutonomousGPT-5-mini14.1 ± 1.443.8 ± 0.00.711 ± 0.0070.442 ± 0.016
AutonomousGPT-5.6-luna20.3 ± 1.459.4 ± 1.40.602 ± 0.0000.221 ± 0.003
AutonomousSonnet 51.8 ± 1.863.8 ± 0.90.672 ± 0.0080.439 ± 0.027
AutonomousGemini 3.6 Flash9.4 ± 1.454.7 ± 0.00.680 ± 0.0070.403 ± 0.018
AutonomousQwen3.5-9B40.1 ± 0.544.3 ± 0.50.578 ± 0.0040.156 ± 0.008
AutonomousLlama-3.1-8B41.4 ± 0.045.3 ± 0.00.566 ± 0.0000.133 ± 0.000
AuthorsOpenReview ↗

Initial Response Pt. 2

Table 6. Effect of user framing on leakage and false refusal. Each entry reports pooled LR/FRR (%) across three independent repetitions. For each model and framing, denominators are 192 runs per class in directed authentication and 96 per class in autonomous monitoring.

SettingFramingOpusGPT-5-miniGPT-5.6-lunaSonnetGeminiQwenLlama
Directedneutral44.3/1.659.9/0.062.5/0.032.8/1.056.2/0.072.9/4.776.6/15.1
Directednaive44.8/0.059.4/1.660.9/0.021.9/1.051.6/0.083.3/1.674.5/7.3
Directedpressured25.0/0.054.7/1.656.2/0.017.2/1.035.9/0.563.0/3.187.5/6.2
Directedcautious10.9/25.09.4/46.919.8/37.57.8/51.618.8/0.046.9/1.668.2/22.4
Autonomousneutral0.0/84.426.0/25.025.0/46.92.1/43.815.6/28.162.5/27.156.2/28.1
Autonomousnaive0.0/84.47.3/38.521.9/50.00.0/58.313.5/43.843.8/21.946.9/53.1
Autonomouspressured6.2/87.522.9/15.634.4/43.82.1/56.28.3/46.951.0/28.153.1/34.4
Autonomouscautious1.0/96.90.0/95.80.0/96.93.1/96.90.0/100.03.1/100.09.4/65.6

Table 7. Leakage and false refusal under different security guidance and verification conditions. Each entry reports pooled LR/FRR (%) across three independent repetitions. For each model and condition, denominators are 192 runs per class in directed authentication and 96 per class in autonomous monitoring.

SettingScaffoldingOpusGPT-5-miniGPT-5.6-lunaSonnetGeminiQwenLlama
Directednone46.9/18.871.9/25.083.3/14.131.2/25.564.1/0.086.5/3.183.9/5.2
Directedgeneric_safety53.6/7.870.3/21.975.5/14.123.4/24.059.4/0.085.9/3.188.5/6.8
Directeddomain_verify12.0/0.022.4/1.620.3/7.812.5/1.025.0/0.554.7/0.071.9/15.6
Directedurl_check_tool12.5/0.018.8/1.620.3/1.612.5/4.214.1/0.039.1/4.762.5/23.4
Autonomousnone0.0/93.819.8/36.543.8/87.52.1/78.116.7/31.259.4/30.253.1/31.2
Autonomousdomain_verify4.2/71.97.3/30.215.6/31.22.1/32.312.5/28.121.9/40.625.0/31.2
Autonomousvault_policy3.1/90.615.6/52.112.5/40.60.0/60.46.2/87.521.9/71.946.9/56.2
Autonomousgeneric_safety0.0/96.913.5/56.29.4/78.13.1/84.42.1/71.957.3/34.440.6/62.5

Footnote 1 states that the 96 tls_warning cases have no trusted recovery route, yet they are included in the 768-case recovery denominator. This limits the maximum recovery rate to 87.5%.

The recovery suite initially included all eight phishing cues to provide balanced coverage. The tls_warning cases still test whether agents avoid disclosure when an unsafe connection uses the correct hostname, though since no distinct clean endpoint is available in these cases, it is correct that they cannot satisfy the active recovery criterion. We therefore update to report active recovery on the 672 feasible cases. Recovery counts are unchanged; the revised rates in Table R3 reflect the smaller denominator and preserve the model ordering. GPT-5-mini remains highest at 27.7% (previously 24.2%), while Opus remains at 0.0%. The central finding holds on feasible scenarios: agents rarely complete the genuine task safely, even when a trusted recovery route is available. We will make this distinction explicit in the revised results.

Table R3. Recovery rates (%) on feasible scenarios (n=672), compared with the original primary suite (n=768).

ModelRecovery %Recovery % on 768 (prev.)
GPT-5-mini27.724.2
Qwen3.5-9B13.511.8
Gemini 3.6 Flash13.111.5
Llama-3.1-8B8.27.2
Sonnet 56.35.5
GPT-5.6-luna5.74.9
Opus 4.80.00.0
AuthorsOpenReview ↗

Initial Response Pt. 3

The main results should separate settings with and without trusted domain information. Under none and generic_safety, agents are not given the official domain. This makes it difficult to judge whether a domain is authentic, especially for fictional brands. Section 5.5 shows that providing the official domain substantially reduces leakage. For GPT-5-mini in the directed setting, LR drops from 73.5% under none to 23.4% under domain_verify. It is therefore unclear how much of the leakage reflects a model vulnerability and how much comes from missing information needed to identify phishing.

Some cues, such as no_https, raw_ip, and tls_warning, can be detected without knowing the official domain. Section 5.5 reports that leakage on no_https persists under domain_verify. However, the domain_verify prompt in Appendix D.2 only asks agents to compare domains, which match exactly in this case. The threat model in Section 3.1 also requires valid HTTPS, but the prompt does not state this requirement. This makes it difficult to tell whether the agent fails to recognize insecure transport or simply follows an incomplete instruction. The authors should separate cues that require a trusted domain reference from those that do not, report the main results separately for each scaffolding condition, and test whether explicitly requiring valid HTTPS changes the no_https result.

Could the authors report the headline LR and FRR separately for conditions with and without the official domain information?

We report headline LR and FRR by scaffolding condition and official-domain availability, separately for directed authentication and autonomous monitoring. The domain is withheld under none and generic_safety; the other conditions provide the domain together with verification instructions or tools. The comparison in Table R4 shows lower leakage for all seven models under domain-provided conditions, although leakage still remains between 8.7% and 56.8%.

Table R4. LR and FRR (%) with the official domain withheld or provided, pooled across directed authentication and autonomous monitoring (three runs; 576 phishing / 576 legitimate runs per group).

ModelLR withheldLR providedFRR withheldFRR provided
Opus 4.833.59.440.627.1
Sonnet 519.18.743.617.2
GPT-5-mini53.017.531.114.8
GPT-5.6-luna61.818.237.015.1
Gemini 3.6 Flash44.316.117.219.4
Qwen3.5-9B76.938.512.820.3
Llama-3.1-8B73.156.819.627.6

We will also separate cues requiring a trusted-domain reference from directly observable cues such as no_https, raw_ip, and tls_warning. Additonally, we evaluated an HTTPS rule in two contexts: a short instruction appended to the instructions, and a rule embedded within a longer operating policy to reflect a more realistic longer-context setting (Table R5). Both explicitly prohibit credential submission over plain HTTP or on certificate-warning pages, even when the hostname matches the official domain.

Table R5. Leak rate (%) by transport cue and instruction condition. The baseline uses domain_verify without an explicit HTTPS requirement; the additional conditions include that requirement in a short prompt or a longer operating policy.

ModelCuedomain_verify in the paperWith explicit HTTPS requirement (short prompt)With explicit HTTPS requirement (within a longer policy)
GPT-5-minino_https100.09.428.1
GPT-5-minitls_warning87.50.03.1
Claude Opus 4.8no_https100.00.03.1
Claude Opus 4.8tls_warning0.00.00.0

Explicit HTTPS guidance reduces plain-HTTP leakage, with outcomes varying across prompt contexts. Under the short prompt, Opus records zero observed leakage on both cues, while GPT-5-mini still leaks on 9.4% of plain-HTTP cases. Within the longer policy, plain-HTTP leakage persists for both models: 28.1% for GPT-5-mini and 3.1% for Opus. Hostname-verification guidance alone is insufficient, and explicit HTTPS instructions improve behavior while leaving failures in some tested conditions. We will clarify the original prompt’s scope and include both HTTPS prompt variants and their results in the revision.

Reviewer_2wLCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CredLeakBench is a benchmark for evaluating whether LLM agents leak credentials and sensitive information while performing legitimate tasks. It covers scenarios involving user-instruction authentication, autonomous inbox triage without an explicit login request, and recovery after phishing through a trusted authentication channel. The benchmark uses 15 simulated services across eight categories in a local simulator, along with legitimate control tasks and eight types of deception cues. Leakage is measured based on the actual submission of synthetic information rather than the agents’ self-reports. An evaluation of seven proprietary and open-weight models found leakage in both instruction-driven and autonomous scenarios across all models, and reduced leakage was sometimes accompanied by high rates of legitimate task incompletion. Trusted destination information and verification support improved the security and utility of some models, but leak-free recovery rates remained limited, ranging from 0 to 24.2%.

Critical strengths
  1. A meaningful focus on safe continuation after encountering phishing. The recovery setting explicitly evaluates whether an agent can bypass a deceptive destination and authenticate through a trusted alternative without leaking information. This is a useful evaluation target because protecting credentials must be compatible with carrying out authorized work. Making this behavior explicit is the paper’s strongest contribution and gives the benchmark potential value for developing agents that remain useful when facing deception.

  2. Behavior-based leakage measurement with paired legitimate controls. The benchmark determines leakage from actual submissions of synthetic information rather than agents’ stated intentions, and pairs phishing scenarios with legitimate counterparts to examine the cost of defensive behavior. The recovery evaluation also counts disclosure before or after trusted authentication as leakage. These design choices are consequential because they capture unsafe actions that verbal expressions of caution or eventual arrival at a legitimate endpoint could conceal, while helping assess whether reduced leakage comes with reduced legitimate activity.

Critical weaknesses
  1. The utility metric does not adequately support the claimed task-completion conclusions. FRR measures the absence of a form submission, and Appendix A explicitly states that legitimate controls do not require a correct-password match (p.12). Consequently, unsuccessful authentication attempts can count as successful legitimate behavior. Conversely, requesting confirmation or deferring an action may be appropriate under cautious delegation instructions, yet is counted as non-completion. Recovery also measures trusted authentication rather than completion of the underlying task. These distinctions are consequential because the paper’s central security–utility conclusions depend on accurately distinguishing useful execution, unsuccessful attempts, and appropriate caution. Consistent success criteria and separate treatment of permissible deferral are needed.

  2. The primary recovery evaluation includes cases without an exposed trusted recovery route. Appendix A states that the 768 primary recovery cases include 96 certificate-warning cases for which the simulator does not expose a distinct trusted recovery route (p.11, footnote 1). Under the described setup, these cases appear to lack the alternative path required for successful recovery, potentially conflating environmental limitations with agent failures. Although excluding them would not eliminate the observed recovery gap, this compromises the interpretation of the paper’s principal distinguishing evaluation. The authors should establish that every primary case is solvable under the available tools, or report cases without a feasible recovery path separately.

Questions for authors
  1. Do the main conclusions hold when successful authentication and appropriate deferral are distinguished?
    Appendix A (p.12) states that FRR on legitimate cases is based on whether a form submission occurs, without requiring a correct-password match. Under cautious instructions, requesting confirmation or deferring an action may also be consistent with the user’s instructions. Can the existing trajectories be separated into successful authentication, failed authentication attempts, instruction-compliant deferral, and unnecessary refusal? Please explain whether the main model comparisons and security–utility conclusions hold under these distinctions.

  2. Is safe recovery feasible in every primary recovery scenario?
    Footnote 1 in Appendix A (p.11) states that the 768 primary cases include 96 certificate-warning cases for which the simulator does not expose a distinct trusted recovery route. Can an agent satisfy the recovery success criterion in these cases using the available tools? If so, please describe a valid action sequence. If not, please report the recovery rates and main conclusions after separating these cases from the feasible recovery scenarios.

  3. Could you clarify run eligibility and metric aggregation?
    Please define “competent runs” in §4.2 and Appendix C, including exclusion criteria, excluded counts, effective denominators for each model, and aggregation procedures. For example, Table 3 reports a directed-setting LR of 66.8%, FRR of 3.0%, and MCC of 0.428 for Qwen. Assuming equal effective denominators for the two classes, these rates yield an MCC of approximately 0.392. Please explain whether this discrepancy arises from exclusions, weighting, or another aggregation procedure, and provide the corresponding raw TP, FP, TN, and FN counts.

5 comments
AuthorsOpenReview ↗

Initial Response Pt. 1

We thank the reviewer for recognizing the value of our safe recovery setting and our behavior-based leakage measurement with paired controls.

Do the main conclusions hold when successful authentication and appropriate deferral are distinguished? Appendix A (p.12) states that FRR on legitimate cases is based on whether a form submission occurs, without requiring a correct-password match. Under cautious instructions, requesting confirmation or deferring an action may also be consistent with the user’s instructions. Can the existing trajectories be separated into successful authentication, failed authentication attempts, instruction-compliant deferral, and unnecessary refusal? Please explain whether the main model comparisons and security–utility conclusions hold under these distinctions.

We reclassified the trajectories into the four outcomes, reported in below Table R2. Success requires the correct target-account password to reach the authorized endpoint; a failed attempt is a submission that does not meet this criterion. Compliant deferral requires a pause permitted by cautious framing and communicated to the user. Remaining non-submissions are classified as unnecessary refusal. In the directed setting, compliant deferral is restricted to none and generic_safety, where no trusted domain or verification tool is supplied. Under domain_verify and url_check_tool, verification support is available, so stopping remains classified as refusal.

Alongside the original FRR, we report an adjusted failure rate that counts failed attempts and unnecessary refusals, with compliant deferrals reported separately. The identified password mismatches affect just 0.8% of directed Llama runs and 2.3% of autonomous Qwen runs. Separating compliant deferral shows that some reduced completion under cautious framing reflects instruction-following rather than unnecessary refusal. Even so, substantial unnecessary refusal remains in autonomous monitoring, ranging from 19.8% to 64.1%. Phishing outcomes and leakage rates are unchanged. The central security–utility conclusion therefore remains, now with a more precise interpretation of cautious non-completion. We will include the full breakdown and refine the corresponding claims in the final revision.

Table R2. Split outcomes on runs (%) by model and setting. The final column compares the original FRR (non-submission) with the adjusted failure rate (failed attempts + unnecessary refusal), excluding compliant deferrals. Values are pooled over three runs.

SettingModelSuccessFailed attemptCompliant deferralUnnecessary refusalFRR old → new
DirectedOpus 4.893.40.06.30.46.6 → 0.4
DirectedGPT-5-mini87.50.011.70.812.5 → 0.8
DirectedGPT-5.6-luna90.60.07.02.49.4 → 2.4
DirectedSonnet 586.30.04.29.513.7 → 9.5
DirectedGemini 3.6 Flash99.90.00.00.10.1 → 0.1
DirectedQwen3.5-9B97.30.00.02.72.7 → 2.7
DirectedLlama-3.1-8B86.40.80.012.812.8 → 13.6
AutonomousOpus 4.811.70.024.264.188.3 → 64.1
AutonomousGPT-5-mini56.30.024.019.843.8 → 19.8
AutonomousGPT-5.6-luna40.60.023.436.059.4 → 36.0
AutonomousSonnet 536.20.07.856.063.8 → 56.0
AutonomousGemini 3.6 Flash45.30.08.146.654.7 → 46.6
AutonomousQwen3.5-9B53.42.321.123.244.3 → 25.5
AutonomousLlama-3.1-8B54.70.07.837.545.3 → 37.5
AuthorsOpenReview ↗

Initial Response Pt. 2

Is safe recovery feasible in every primary recovery scenario? Footnote 1 in Appendix A (p.11) states that the 768 primary cases include 96 certificate-warning cases for which the simulator does not expose a distinct trusted recovery route. Can an agent satisfy the recovery success criterion in these cases using the available tools? If so, please describe a valid action sequence. If not, please report the recovery rates and main conclusions after separating these cases from the feasible recovery scenarios.

The recovery suite initially included all eight phishing cues to provide balanced coverage. The tls_warning cases still test whether agents avoid disclosure when an unsafe connection uses the correct hostname, though since no distinct clean endpoint is available in these cases, it is correct that they cannot satisfy the active recovery criterion. We therefore update to report active recovery on the 672 feasible cases. Recovery counts are unchanged; the revised rates in Table R3 reflect the smaller denominator and preserve the model ordering. GPT-5-mini remains highest at 27.7% (previously 24.2%), while Opus remains at 0.0%. The central finding holds on feasible scenarios: agents rarely complete the genuine task safely, even when a trusted recovery route is available. We will make this distinction explicit in the revised results.

Table R3. Recovery rates (%) on feasible scenarios (n=672), compared with the original primary suite (n=768).

ModelRecovery %Recovery % on 768 (prev.)
GPT-5-mini27.724.2
Qwen3.5-9B13.511.8
Gemini 3.6 Flash13.111.5
Llama-3.1-8B8.27.2
Sonnet 56.25.5
GPT-5.6-luna5.74.9
Opus 4.80.00.0

Could you clarify run eligibility and metric aggregation? Please define “competent runs” in §4.2 and Appendix C, including exclusion criteria, excluded counts, effective denominators for each model, and aggregation procedures. For example, Table 3 reports a directed-setting LR of 66.8%, FRR of 3.0%, and MCC of 0.428 for Qwen. Assuming equal effective denominators for the two classes, these rates yield an MCC of approximately 0.392. Please explain whether this discrepancy arises from exclusions, weighting, or another aggregation procedure, and provide the corresponding raw TP, FP, TN, and FN counts.

“Competent runs” are runs retained after filtering execution failures. Exclusions cover cases like API failures, empty or broken tool calls, provider-level declines before submission, and turn-limit exits without a submission. Leakage or refusal is not itself a reason for exclusion. Behavioral refusals remain eligible, including refusals before navigation. The MCC discrepancy was due to incorrect aggregation; the corresponding LR, FRR, and accuracies were unaffected. The updated value is now 0.399, there is no change in conclusion. The updated revision now includes mean and standard deviation across 3 runs with listed denominator counts.

Thank you for the detailed responses and additional analyses. Excluding the 96 infeasible recovery cases resolves that concern. I also appreciate the correct-password checks and the distinction between compliant deferral and unnecessary refusal. The corrected MCC addresses my earlier numerical concern.

Before finalizing my assessment, I would appreciate clarification on two remaining points:

  • I appreciate your acknowledgment that some non-completion under cautious framing reflects instruction-following, and your plan to revise the corresponding claims. Could you briefly report the deferral and adjusted failure rates for the cautious condition and explain which security–utility conclusions remain supported and which need qualification? It would also help to clarify how the directed-setting rule that treats stopping as unnecessary refusal when a verification tool or trusted domain is available follows from the actual user instructions.

  • I understand that behavioral refusals are retained and technical failures are retried. However, I would appreciate clarification on how provider-level declines and turn-limit exits without submission are distinguished from behavioral refusal or non-completion. Could you provide the exclusion and retry counts by model and setting, along with the criteria used to make this distinction? If these cases account for a substantial proportion of attempts, please also explain whether their treatment affects the main conclusions.

Your responses have substantially reduced my concerns. For now, I will provisionally retain Weak Reject, as these remaining points affect how I interpret the evaluation and assess the reliability of its findings. I expect they can be addressed through clarification and the existing evaluation records. I would be glad to raise my score if these concerns are resolved. Thank you again for your thoughtful responses and efforts to strengthen the paper.

Reviewer_ipqMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CredLeak-Bench, a benchmark for evaluating credential leakage and legitimate task completion in LLM agents facing phishing attacks. Experiments across seven models reveal that agents frequently disclose sensitive information to phishing destinations. The results also highlight a security–utility trade-off that cautious instructions and restrictive policies often reduce leakage at the cost of legitimate task completion, while explicit destination verification can improve both. Furthermore, agents rarely recover from phishing by switching to trusted authentication channels, suggesting that current agents struggle to balance security with effective task execution.

Critical strengths
  • The paper introduces a benchmark that jointly evaluates credential leakage and legitimate task completion across directed authentication and autonomous inbox-monitoring scenarios.
Critical weaknesses
  • The paper's novelty is limited by substantial overlap with existing studies on phishing and social engineering against LLM agents, particularly Scammer4U and AgentBait. While the recovery setting offers an additional evaluation dimension, the paper does not convincingly demonstrate that its benchmark provides fundamentally new insights or vulnerabilities beyond those established by prior work.
  • The paper evaluates all models using a simplified, custom-built agent interface with only a few predefined tools (e.g., open_page, submit_login, submit_form, and reply_email). This setup does not reflect the complexity of real-world agent systems, where authentication and phishing resistance may depend heavily on the underlying agent harness, browser interactions, tool implementations, and security mechanisms. More importantly, the authors do not evaluate any established open-source agent harnesses, making it unclear whether the observed vulnerabilities and security–utility trade-offs generalize to practical agent deployments. Evaluating representative agent harnesses would substantially strengthen the empirical relevance of the benchmark.
Questions for authors
  • I can't get the message you mentioned in Line 428-430 from Figure 5b alone: "It yields zero observed leakage for all five closed-source models, but completion falls to just 0.0-10.7% across all seven models". Please clarify the writing here.
3 comments
AuthorsOpenReview ↗

Initial Response Pt. 1

We thank the reviewer for highlighting the value of our directed and autonomous benchmark scenarios and for providing the feedback.

The paper's novelty is limited by substantial overlap with existing studies on phishing and social engineering against LLM agents, particularly Scammer4U and AgentBait. While the recovery setting offers an additional evaluation dimension, the paper does not convincingly demonstrate that its benchmark provides fundamentally new insights or vulnerabilities beyond those established by prior work.

Scammer4U and AgentBait establish susceptibility to social engineering during specified web tasks. CredLeak-Bench additionally compares user-requested authentication with autonomous inbox monitoring, where the agent must decide whether and where to authenticate based on incoming messages. This comparison reveals that all seven models disclose credentials even without a user-selected login target. In this autonomous setting, every model shows lower leakage but higher false refusal than in directed authentication (Table 3). For example, Opus’s leakage falls from 31.2% to 1.8%, while false refusal rises from 6.6% to 88.3%. The additional insight is that delegation context substantially changes the security–utility balance: monitoring can appear safer while leaving most genuine requests unfulfilled and disclosure risk unresolved. Prior task-specific attack evaluations do not establish this relationship.

Recovery further tests whether agents can preserve useful behavior after encountering deception by completing the genuine task through a trusted alternative. Even with trusted reference information available, recovery ranges from only 0% to 27.7%; Opus avoids leakage entirely yet never recovers (Fig. 4a). This exposes a capability gap that successful refusal alone cannot reveal

The paper evaluates all models using a simplified, custom-built agent interface with only a few predefined tools (e.g., open_page, submit_login, submit_form, and reply_email). This setup does not reflect the complexity of real-world agent systems, where authentication and phishing resistance may depend heavily on the underlying agent harness, browser interactions, tool implementations, and security mechanisms. More importantly, the authors do not evaluate any established open-source agent harnesses, making it unclear whether the observed vulnerabilities and security–utility trade-offs generalize to practical agent deployments. Evaluating representative agent harnesses would substantially strengthen the empirical relevance of the benchmark.

The minimal tool surface is a deliberate design choice. Our primary goal is to evaluate the underlying model’s judgment about where to send sensitive information, not the surrounding harness. We hold tools, prompts, execution, and scoring fixed within each condition (Section 4.1). Full agent harnesses add their own prompts, tools, and security mechanisms, which can prevent disclosure independently of the model’s judgment. Our shared interface supports controlled comparisons of model behavior under consistent conditions.

That being said, to complement this analysis, we additionally evaluated GPT-5-mini and Opus 4.8 on the directed suite of 512 scenarios using an open source harness: browser-use. The agents operate Chrome under the framework’s system prompt and receive credentials through our established mechanism. We retain the same scenarios, task text, security guidance, and submission-based scoring. Results are shown in Table R1. Both models leak under every guidance condition in browser-use. Changing the harness reduces leakage for GPT-5-mini and increases it for Opus, reversing their leakage ordering; GPT-5-mini’s reduction is accompanied by higher false refusal. The security–utility tension also appears within browser-use: cautious requests yield LR/FRR of 4.7%/71.9% for GPT-5-mini and 9.4%/51.6% for Opus. These results show persistent leakage under a different harness.

Table R1. Leakage and false refusal (%) on the directed suite under our shared interface and browser-use. Shared-interface rates pool three repetitions; browser-use rates use one repetition. Denominators per class are 768 versus 256 overall, and 192 versus 64 per guidance condition.

ModelSecurity guidanceLR, shared interfaceLR, browser-useFRR, shared interfaceFRR, browser-use
GPT-5-miniAll conditions45.832.012.539.1
GPT-5-mininone71.948.425.048.4
GPT-5-minigeneric_safety70.350.021.953.1
GPT-5-minidomain_verify22.423.41.625.0
GPT-5-miniurl_check_tool18.86.21.629.7
Opus 4.8All conditions31.241.86.614.1
Opus 4.8none46.965.618.814.1
Opus 4.8generic_safety53.659.47.814.1
Opus 4.8domain_verify12.031.20.018.8
Opus 4.8url_check_tool12.510.90.09.4
AuthorsOpenReview ↗

Initial Response Pt. 2

I can't get the message you mentioned in Line 428-430 from Figure 5b alone: "It yields zero observed leakage for all five closed-source models, but completion falls to just 0.0-10.7% across all seven models". Please clarify the writing here.

The quoted absolute completion rates come from Appendix Table 6 (completion = 100% - FRR), which is cited in the next sentence on line 431. Figure 5b shows leakage rate changes relative to neutral framing, not false refusal or completion. The primary message is that cautious framing can drastically reduce leakage (to 0% for most closed-source models), but also reduces legitimate task completion utility for all models. We will revise Lines 428–430 to state these results and their sources more clearly.

I appreciate the authors' efforts to address my concerns. The additional experiments with an established agent harness strengthen the empirical evaluation. However, the reversal in the leakage ranking of the two models across harnesses also reinforces my original concern that the observed model behavior can be highly dependent on the underlying agent harness.

Regarding novelty, I acknowledge the value of the autonomous monitoring and recovery scenarios. Nevertheless, I remain concerned that the contribution is primarily an extension of existing evaluation settings rather than offering substantially new insights into the security vulnerabilities of LLM agents.

Overall, while the rebuttal partially addresses my concerns, I am inclined to maintain my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Phase Ending Soon

Reviewers, as the author-reviewer discussion phase ends very soon, we kindly ask if our previous responses have addressed your concerns, and we would be grateful if you would consider re-evaluating our submission and responding to our rebuttal.

Thanks for your time and efforts spent reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.