- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CredLeak-Bench, a sandboxed benchmark that measures credential leakage in LLM agents through actual form submissions rather than model self-reports. Its core contribution is the joint measurement of leak rate and false-refusal rate via paired phishing/legitimate scenarios across directed and autonomous settings, plus a recovery setting evaluates whether an agent can bypass a phishing link and complete the genuine task through a trusted path.
Critical strengths
-
The measurement is reliable. Each phishing scenario has a legitimate counterpart that differs only in origin. Leakage is verified by matching submitted values against vault credentials in captured network logs, providing more reliable evidence than model self-reports.
-
A useful recovery setting. As mentioned by the author, avoiding leakage is not the same as completing a task safely. Table 13 shows that Qwen3.5-9B can leak credentials even after reaching a trusted endpoint. This distinction matters because reaching the correct destination does not undo earlier leakage or ensure that subsequent actions are safe.
Critical weaknesses
- The paper needs clearer reporting to support its comparisons. For example, the autonomous setting includes only 16 phishing cases per deception cue (Appendix B) and 32 legitimate cases per framing. The Reproducibility Statement says that aggregate rates are stable across reruns, but no results are provided to support this claim. The number of runs per scenario is also unclear. For example, Table 6 reports an autonomous cautious FRR of 99.3% for Sonnet 5. This value would not be possible if each of the 32 legitimate scenarios was run once and FRR was calculated directly over these scenarios. The authors should clarify the number of runs and the denominator used.
There are also several unclear points in the reported metrics. LR and FRR are calculated over “competent” runs, but the paper does not clearly define competence or report how many runs are excluded. Footnote 1 states that the 96 tls_warning cases have no trusted recovery route, yet they are included in the 768-case recovery denominator. This limits the maximum recovery rate to 87.5%. The aggregation in Table 3 also needs explanation. In the directed setting, its values differ from the unweighted means of the values in Tables 6 and 7: Sonnet 5 has LR of 31.5% versus 30.8% and FRR of 26.6% versus 25.2%; Gemini 3.6 Flash has LR of 53.1% versus 53.8%. The authors should explain how these results are calculated and provide evidence of their variation across runs.
- The main results should separate settings with and without trusted domain information. Under none and generic_safety, agents are not given the official domain. This makes it difficult to judge whether a domain is authentic, especially for fictional brands. Section 5.5 shows that providing the official domain substantially reduces leakage. For GPT-5-mini in the directed setting, LR drops from 73.5% under none to 23.4% under domain_verify. It is therefore unclear how much of the leakage reflects a model vulnerability and how much comes from missing information needed to identify phishing.
Some cues, such as no_https, raw_ip, and tls_warning, can be detected without knowing the official domain. Section 5.5 reports that leakage on no_https persists under domain_verify. However, the domain_verify prompt in Appendix D.2 only asks agents to compare domains, which match exactly in this case. The threat model in Section 3.1 also requires valid HTTPS, but the prompt does not state this requirement. This makes it difficult to tell whether the agent fails to recognize insecure transport or simply follows an incomplete instruction. The authors should separate cues that require a trusted domain reference from those that do not, report the main results separately for each scaffolding condition, and test whether explicitly requiring valid HTTPS changes the no_https result.
Questions for authors
- How do the authors define a “competent” run, and how many runs were excluded for each model and setting?
- Could the authors report the headline LR and FRR separately for conditions with and without the official domain information?
- In the recovery setting, the headline recovery rate groups escalation to the user with other non-completion outcomes. For example, Opus 4.8 escalates in 722 of 768 cases (Table 13) and is described as never recovering. Why is escalation not credited as a safe outcome, and how would the headline results change if it were?
6 comments
Initial Response Pt. 2
Table 6. Effect of user framing on leakage and false refusal. Each entry reports pooled LR/FRR (%) across three independent repetitions. For each model and framing, denominators are 192 runs per class in directed authentication and 96 per class in autonomous monitoring.
| Setting | Framing | Opus | GPT-5-mini | GPT-5.6-luna | Sonnet | Gemini | Qwen | Llama |
|---|---|---|---|---|---|---|---|---|
| Directed | neutral | 44.3/1.6 | 59.9/0.0 | 62.5/0.0 | 32.8/1.0 | 56.2/0.0 | 72.9/4.7 | 76.6/15.1 |
| Directed | naive | 44.8/0.0 | 59.4/1.6 | 60.9/0.0 | 21.9/1.0 | 51.6/0.0 | 83.3/1.6 | 74.5/7.3 |
| Directed | pressured | 25.0/0.0 | 54.7/1.6 | 56.2/0.0 | 17.2/1.0 | 35.9/0.5 | 63.0/3.1 | 87.5/6.2 |
| Directed | cautious | 10.9/25.0 | 9.4/46.9 | 19.8/37.5 | 7.8/51.6 | 18.8/0.0 | 46.9/1.6 | 68.2/22.4 |
| Autonomous | neutral | 0.0/84.4 | 26.0/25.0 | 25.0/46.9 | 2.1/43.8 | 15.6/28.1 | 62.5/27.1 | 56.2/28.1 |
| Autonomous | naive | 0.0/84.4 | 7.3/38.5 | 21.9/50.0 | 0.0/58.3 | 13.5/43.8 | 43.8/21.9 | 46.9/53.1 |
| Autonomous | pressured | 6.2/87.5 | 22.9/15.6 | 34.4/43.8 | 2.1/56.2 | 8.3/46.9 | 51.0/28.1 | 53.1/34.4 |
| Autonomous | cautious | 1.0/96.9 | 0.0/95.8 | 0.0/96.9 | 3.1/96.9 | 0.0/100.0 | 3.1/100.0 | 9.4/65.6 |
Table 7. Leakage and false refusal under different security guidance and verification conditions. Each entry reports pooled LR/FRR (%) across three independent repetitions. For each model and condition, denominators are 192 runs per class in directed authentication and 96 per class in autonomous monitoring.
| Setting | Scaffolding | Opus | GPT-5-mini | GPT-5.6-luna | Sonnet | Gemini | Qwen | Llama |
|---|---|---|---|---|---|---|---|---|
| Directed | none | 46.9/18.8 | 71.9/25.0 | 83.3/14.1 | 31.2/25.5 | 64.1/0.0 | 86.5/3.1 | 83.9/5.2 |
| Directed | generic_safety | 53.6/7.8 | 70.3/21.9 | 75.5/14.1 | 23.4/24.0 | 59.4/0.0 | 85.9/3.1 | 88.5/6.8 |
| Directed | domain_verify | 12.0/0.0 | 22.4/1.6 | 20.3/7.8 | 12.5/1.0 | 25.0/0.5 | 54.7/0.0 | 71.9/15.6 |
| Directed | url_check_tool | 12.5/0.0 | 18.8/1.6 | 20.3/1.6 | 12.5/4.2 | 14.1/0.0 | 39.1/4.7 | 62.5/23.4 |
| Autonomous | none | 0.0/93.8 | 19.8/36.5 | 43.8/87.5 | 2.1/78.1 | 16.7/31.2 | 59.4/30.2 | 53.1/31.2 |
| Autonomous | domain_verify | 4.2/71.9 | 7.3/30.2 | 15.6/31.2 | 2.1/32.3 | 12.5/28.1 | 21.9/40.6 | 25.0/31.2 |
| Autonomous | vault_policy | 3.1/90.6 | 15.6/52.1 | 12.5/40.6 | 0.0/60.4 | 6.2/87.5 | 21.9/71.9 | 46.9/56.2 |
| Autonomous | generic_safety | 0.0/96.9 | 13.5/56.2 | 9.4/78.1 | 3.1/84.4 | 2.1/71.9 | 57.3/34.4 | 40.6/62.5 |
Footnote 1 states that the 96 tls_warning cases have no trusted recovery route, yet they are included in the 768-case recovery denominator. This limits the maximum recovery rate to 87.5%.
The recovery suite initially included all eight phishing cues to provide balanced coverage. The tls_warning cases still test whether agents avoid disclosure when an unsafe connection uses the correct hostname, though since no distinct clean endpoint is available in these cases, it is correct that they cannot satisfy the active recovery criterion. We therefore update to report active recovery on the 672 feasible cases. Recovery counts are unchanged; the revised rates in Table R3 reflect the smaller denominator and preserve the model ordering. GPT-5-mini remains highest at 27.7% (previously 24.2%), while Opus remains at 0.0%. The central finding holds on feasible scenarios: agents rarely complete the genuine task safely, even when a trusted recovery route is available. We will make this distinction explicit in the revised results.
Table R3. Recovery rates (%) on feasible scenarios (n=672), compared with the original primary suite (n=768).
| Model | Recovery % | Recovery % on 768 (prev.) |
|---|---|---|
| GPT-5-mini | 27.7 | 24.2 |
| Qwen3.5-9B | 13.5 | 11.8 |
| Gemini 3.6 Flash | 13.1 | 11.5 |
| Llama-3.1-8B | 8.2 | 7.2 |
| Sonnet 5 | 6.3 | 5.5 |
| GPT-5.6-luna | 5.7 | 4.9 |
| Opus 4.8 | 0.0 | 0.0 |
Initial Response Pt. 3
The main results should separate settings with and without trusted domain information. Under none and generic_safety, agents are not given the official domain. This makes it difficult to judge whether a domain is authentic, especially for fictional brands. Section 5.5 shows that providing the official domain substantially reduces leakage. For GPT-5-mini in the directed setting, LR drops from 73.5% under none to 23.4% under domain_verify. It is therefore unclear how much of the leakage reflects a model vulnerability and how much comes from missing information needed to identify phishing.
Some cues, such as no_https, raw_ip, and tls_warning, can be detected without knowing the official domain. Section 5.5 reports that leakage on no_https persists under domain_verify. However, the domain_verify prompt in Appendix D.2 only asks agents to compare domains, which match exactly in this case. The threat model in Section 3.1 also requires valid HTTPS, but the prompt does not state this requirement. This makes it difficult to tell whether the agent fails to recognize insecure transport or simply follows an incomplete instruction. The authors should separate cues that require a trusted domain reference from those that do not, report the main results separately for each scaffolding condition, and test whether explicitly requiring valid HTTPS changes the no_https result.
Could the authors report the headline LR and FRR separately for conditions with and without the official domain information?
We report headline LR and FRR by scaffolding condition and official-domain availability, separately for directed authentication and autonomous monitoring. The domain is withheld under none and generic_safety; the other conditions provide the domain together with verification instructions or tools. The comparison in Table R4 shows lower leakage for all seven models under domain-provided conditions, although leakage still remains between 8.7% and 56.8%.
Table R4. LR and FRR (%) with the official domain withheld or provided, pooled across directed authentication and autonomous monitoring (three runs; 576 phishing / 576 legitimate runs per group).
| Model | LR withheld | LR provided | FRR withheld | FRR provided |
|---|---|---|---|---|
| Opus 4.8 | 33.5 | 9.4 | 40.6 | 27.1 |
| Sonnet 5 | 19.1 | 8.7 | 43.6 | 17.2 |
| GPT-5-mini | 53.0 | 17.5 | 31.1 | 14.8 |
| GPT-5.6-luna | 61.8 | 18.2 | 37.0 | 15.1 |
| Gemini 3.6 Flash | 44.3 | 16.1 | 17.2 | 19.4 |
| Qwen3.5-9B | 76.9 | 38.5 | 12.8 | 20.3 |
| Llama-3.1-8B | 73.1 | 56.8 | 19.6 | 27.6 |
We will also separate cues requiring a trusted-domain reference from directly observable cues such as no_https, raw_ip, and tls_warning. Additonally, we evaluated an HTTPS rule in two contexts: a short instruction appended to the instructions, and a rule embedded within a longer operating policy to reflect a more realistic longer-context setting (Table R5). Both explicitly prohibit credential submission over plain HTTP or on certificate-warning pages, even when the hostname matches the official domain.
Table R5. Leak rate (%) by transport cue and instruction condition. The baseline uses domain_verify without an explicit HTTPS requirement; the additional conditions include that requirement in a short prompt or a longer operating policy.
| Model | Cue | domain_verify in the paper | With explicit HTTPS requirement (short prompt) | With explicit HTTPS requirement (within a longer policy) |
|---|---|---|---|---|
| GPT-5-mini | no_https | 100.0 | 9.4 | 28.1 |
| GPT-5-mini | tls_warning | 87.5 | 0.0 | 3.1 |
| Claude Opus 4.8 | no_https | 100.0 | 0.0 | 3.1 |
| Claude Opus 4.8 | tls_warning | 0.0 | 0.0 | 0.0 |
Explicit HTTPS guidance reduces plain-HTTP leakage, with outcomes varying across prompt contexts. Under the short prompt, Opus records zero observed leakage on both cues, while GPT-5-mini still leaks on 9.4% of plain-HTTP cases. Within the longer policy, plain-HTTP leakage persists for both models: 28.1% for GPT-5-mini and 3.1% for Opus. Hostname-verification guidance alone is insufficient, and explicit HTTPS instructions improve behavior while leaving failures in some tested conditions. We will clarify the original prompt’s scope and include both HTTPS prompt variants and their results in the revision.
Initial Response Pt. 1
We thank the reviewer for their comprehensive review and for acknowledging the usefulness of our proposed recovery setting.
“Competent” refers to a valid evaluation attempt. Exclusions cover cases like API failures, empty or broken tool calls, provider-level declines before submission, and turn-limit exits without a submission. A run is never left out because of what the agent decided, so leaking or refusing has no effect on whether a run counts. Technical failures are retried, and each scenario contributes three valid runs to the final analysis.
We formally re-evaluated the full directed and autonomous suites over three independent repetitions. Table 3 now reports mean ± standard deviation for LR, FRR, Balanced Accuracy, and MCC. Tables 6 and 7 report pooled LR and FRR by framing and scaffolding condition. Each scenario contributes one valid run per repetition. Per repetition, the directed setting contains 256 phishing and 256 legitimate runs per model, and autonomous monitoring contains 128 of each. Across three repetitions, each framing or scaffolding row in Tables 6 and 7 contains 192 runs per class in the directed setting and 96 in autonomous monitoring. Each autonomous deception cue contains 48 phishing runs across repetitions. The standard deviations of LR and FRR in Table 3 range from 0.0 to 1.8 percentage points.
The updated results preserve the main qualitative findings: all seven models disclose information in both settings; cautious framing generally reduces leakage at a cost to legitimate task completion; verification support improves both metrics for most models in directed authentication; and the effects of the autonomous vault policy vary across models. The broad security–utility pattern persists: every model has lower leakage but higher false refusal in autonomous monitoring than in directed authentication.
Table 3. Leakage, false refusal, balanced accuracy, and MCC across models and settings. Values are mean ± standard deviation across three independent repetitions, with metrics computed separately per repetition. LR and FRR are percentages. Each repetition contains 256 phishing and 256 legitimate runs in directed authentication, and 128 of each in autonomous monitoring.