- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines the validity of existing metrics for AI red-teaming, particularly Attack Success Rate. The authors propose a probabilistic framework that defines model safety risk as expected severity under a target threat distribution. They derive an error bound based on Wasserstein distance and partial smoothness, accounting for distributional misalignment, behavioral irregularity, severity assessment bias, and sampling error. Experiments show that ASR varies from 0.118 to 0.905 across evaluation distributions, even with a fixed model and judge. The paper further analyzes safety behavior smoothness using 38,659 red-teaming conversations and proposes complementary evaluation metrics.
Critical strengths
- The paper addresses an important and practical problem concerning the validity of AI red-teaming evaluations.
- The theoretical framework provides a clear decomposition of the factors contributing to risk estimation errors.
- The controlled experiments demonstrate the substantial influence of evaluation distributions on ASR, offering useful insights into current evaluation practices.
Critical weaknesses
- The theoretical contributions rely heavily on established mathematical tools, including Wasserstein distance and Lipschitz continuity, limiting the overall methodological novelty.
- The reported bounds are relatively loose, with tightness of 2.9%–27.6%, and the empirical analysis involves different target models, which does not fully satisfy the theoretical assumptions.
- The proposed metrics lack comprehensive empirical validation, particularly regarding their ability to estimate safety risk more accurately under realistic threat distributions.
Questions for authors
- Could the authors further validate the tightness and practical utility of the theoretical bounds using a fixed target model?
- How sensitive are the results to the choice of target distribution, particularly when realistic threat distributions are difficult to estimate?
- Could the authors provide more direct comparisons demonstrating that the proposed metrics improve risk estimation accuracy over conventional ASR?
2 comments
Additional experiment: fixed-model bound verification on prompt families
Motivation. Our hh-rlhf evidence used prompts answered by different models, so measured smoothness could reflect model differences. Here we fix one model and a set of underlying requests and vary only their phrasing (details in new PDF, Section 5.3 and Appendix G).
Setup. We took 50 harmful requests from WildJailbreak and built a 10-prompt family for each: the original, 3 paraphrases, 3 style rewrites (casual, formal, terse), and 3 of WildJailbreak's own jailbreak versions. Paraphrases and rewrites were generated with Qwen2.5-7B-Instruct, filtered automatically, and reviewed manually for meaning preservation. Mistral-7B-Instruct-v0.3 gave four responses per prompt (one greedy, three sampled), scored by Gemini with a StrongREJECT-style gated rubric (refusals score zero). A prompt's severity is its mean over responses, an estimate of expected severity, normalized so ; we also report compliance. The target is uniform over the 350 non-jailbreak prompts. Evaluation sets mix in a growing share of jailbreak versions (0-100%), restrict to one prompt kind, or restrict to one of two topic clusters. We compute exact and the tightest bound of the Appendix B form by one linear program over and every -Lipschitz , against the trivial bound .
| Score | Jailbreak share | Gap | Bound | Trivial | Alignment share | |
|---|---|---|---|---|---|---|
| Gated severity | 10% | 0.006 | 0.044 | 0.046 | 0.088 | 53% |
| Gated severity | 30% | 0.017 | 0.133 | 0.087 | 0.099 | 42% |
| Compliance | 10% | 0.027 | 0.044 | 0.127 | 0.402 | 63% |
| Compliance | 50% | 0.136 | 0.222 | 0.399 | 0.511 | 54% |
Findings.
- The bound holds for every evaluation distribution and is nontrivial for moderate shifts, with tightness of 12-29% on gated severity and 21-42% on compliance (Figure 6).
- Held-out. Fitting and on half the families and predicting on the other half without refitting, the gap stayed under the prediction in all 320 cases (20 splits), within 0.007 of the in-sample optimum (Table 12).
- The form is far looser, so the revision uses the tighter Appendix B form.
- Irregularity concentrates in jailbreaks. Rewordings of a request differ in severity by 0.031 on average, against 0.092 between a rewording and a jailbreak version.
- Response randomness matters. Using only the greedy response narrows where the bound is nontrivial (Table 14).
- Topic shift. Topic clusters sit at of 0.28-0.36 yet change risk by at most 0.004, so large need not mean large error.
Limitations. One model, low base severity, mostly synthetic prompts, a fixed , and informative bounds only for shifts toward related prompts.
We thank the reviewer for the clear summary and questions. Line numbers refer to the revised PDF.
W1 (novelty). We agree that the mathematical tools are established, and we see this as an advantage for a measurement framework, since every term can be computed with standard software. Our contributions lie in how the tools are applied. First, we define red-teaming success as estimation of a named estimand, the risk under a stated threat model, and decompose the error of any reported metric into alignment, irregularity, severity-validity, and sampling terms (Corollary 3), which shows that collecting more prompts addresses only the last. Second, we introduce a partial-smoothness condition suited to brittle safety behavior, since global Lipschitz continuity fails for models whose behavior flips under adversarial phrasing. Third, we make the terms estimable in practice, including an exact linear-program computation of the tightest bound of this form and, in the revision, a fixed-model validation with held-out prediction (Section 5.3).
W2 / Q1 (tightness with a fixed model). The supplementary experiment below fixes the target model and the underlying requests (Section 5.3, L449-503; Appendix G). The bound holds for every evaluation distribution. With the tighter irregularity term of Appendix B, its tightness is 12-29% on gated severity and 21-42% on compliance wherever it improves on the trivial bound, compared with 2.9-27.6% in the submitted cross-model analysis (Table 1). A held-out check confirms that constants fitted on half the requests predict a valid bound on the other half in every case (Table 12).
W3 / Q3 (comparison with conventional ASR). Our claim for severity-weighted ASR concerns what it estimates rather than how far it moves under distribution shift. Because risk is defined as expected severity, ASRs targets it directly, while flat ASR replaces severity with a binary indicator, which adds to the severity-validity term whenever unsafe outputs differ in severity (Section 4). For example, on qsuccessful flat ASR saturates at 1.000 by construction while ASRs is 0.563 (Section 5.2, L424). Table 6 addresses a different question, how far each metric moves from its own p∗ reference under distribution shift, and there the two behave similarly once their different scales are accounted for (mean relative gaps of 60% and 64%). We revised its caption and the corresponding text in Section 5.2 to make this explicit. More broadly, the framework's contribution to accuracy is diagnostic. Flat ASR on its own gives no indication of how far it may be from the risk under the threat model, whereas W1, the smoothness profile, and judge reliability indicate when an estimate can be trusted. The fixed-model experiment shows this working: with smoothness fitted on half the requests, the bound gave a valid limit on the estimation error for the other half in every held-out case. The importance-weighted estimator (Section 4, Appendix C) is the component designed to correct distribution-shift bias directly, and evaluating it empirically under a realistic threat distribution is an important next step.
Q2 (sensitivity to the target distribution). We have not yet varied p∗ in the new experiments, and we state this as a limitation in Section 5.3 (L497). Our results are defined relative to a stated p∗ by design. The framework makes the dependence on the threat model explicit rather than removing it, and a different p∗ asks a different question about the model. When a realistic threat distribution is hard to estimate, the framework suggests two practical options. First, a threat-model-specific approximation, even a synthetic one such as the user simulator of Section 5.1, can be stated and audited, which improves on an implicit uniform target (Section 3.2, Assumption 2). Second, when several candidate distributions are plausible, one can report the risk estimate and diagnostics under each, or take p∗ to be a family of threat-model distributions and bound the worst case over it, which connects to distributionally robust optimization (Section 6). A systematic sensitivity analysis over alternative choices of p∗ is a natural next step.