acceptodds
Under review as a conference paper at ICLR 2027

Toward Valid Measures of Efficacy and Success for AI Red-teaming

Abstract

Red-teaming has become a cornerstone of GenAI safety evaluation, but the field lacks principled criteria for measuring its efficacy and success. We identify several core challenges that undermine the validity of existing metrics, such as Attack Success Rate (ASR). To understand and address them, we propose a probabilistic framework that grounds red-teaming success in a single well-defined quantity: the safety risk of the model (or expected severity of its outputs) under a target usage distribution p* capturing the threat model of interest. We show that the gap between this quantity and any empirically computed success metric is controlled by the distance between the red-teaming distribution q and the target distribution p*, the smoothness of the underlying severity function outside a small set of irregular prompts, the validity of severity assessments in practice, and a sampling error that shrinks with corpus size. Our formalization offers three core insights: the validity of red-teaming success metrics depends on 1) the alignment of the attack distribution under red-teaming and under the threat model of interest, 2) the empirical assessment of severity matching the real severity function, and 3) the smoothness of model safety behavior. In particular, because more prompts reduce only the sampling error, validity does not necessarily improve with evaluation corpus size or superficial attack diversity. We apply the framework to two corpora. Holding the target model and safety judge fixed and varying only the evaluation distribution, measured ASR ranges from 0.118 to 0.905, and distance from the reference distribution orders the resulting error exactly. On a corpus of 38,659 human red-teaming conversations we then estimate the smoothness pair directly, reporting what are to our knowledge the first empirical estimates of severity smoothness in this literature and showing that preference-trained models exhibit markedly smoother safety behavior than base models. We discuss the implications of our formalization for evaluation practice and outline better approaches for measuring success going forward.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI red-teaming, safety evaluation, attack success rate, distributional alignment, success metrics, severity-weighted metrics, importance sampling, GenAI safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XXs9OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the validity of existing metrics for AI red-teaming, particularly Attack Success Rate. The authors propose a probabilistic framework that defines model safety risk as expected severity under a target threat distribution. They derive an error bound based on Wasserstein distance and partial smoothness, accounting for distributional misalignment, behavioral irregularity, severity assessment bias, and sampling error. Experiments show that ASR varies from 0.118 to 0.905 across evaluation distributions, even with a fixed model and judge. The paper further analyzes safety behavior smoothness using 38,659 red-teaming conversations and proposes complementary evaluation metrics.

Critical strengths
  1. The paper addresses an important and practical problem concerning the validity of AI red-teaming evaluations.
  2. The theoretical framework provides a clear decomposition of the factors contributing to risk estimation errors.
  3. The controlled experiments demonstrate the substantial influence of evaluation distributions on ASR, offering useful insights into current evaluation practices.
Critical weaknesses
  1. The theoretical contributions rely heavily on established mathematical tools, including Wasserstein distance and Lipschitz continuity, limiting the overall methodological novelty.
  2. The reported bounds are relatively loose, with tightness of 2.9%–27.6%, and the empirical analysis involves different target models, which does not fully satisfy the theoretical assumptions.
  3. The proposed metrics lack comprehensive empirical validation, particularly regarding their ability to estimate safety risk more accurately under realistic threat distributions.
Questions for authors
  1. Could the authors further validate the tightness and practical utility of the theoretical bounds using a fixed target model?
  2. How sensitive are the results to the choice of target distribution, particularly when realistic threat distributions are difficult to estimate?
  3. Could the authors provide more direct comparisons demonstrating that the proposed metrics improve risk estimation accuracy over conventional ASR?
2 comments

We thank the reviewer for the clear summary and questions. Line numbers refer to the revised PDF.

W1 (novelty). We agree that the mathematical tools are established, and we see this as an advantage for a measurement framework, since every term can be computed with standard software. Our contributions lie in how the tools are applied. First, we define red-teaming success as estimation of a named estimand, the risk under a stated threat model, and decompose the error of any reported metric into alignment, irregularity, severity-validity, and sampling terms (Corollary 3), which shows that collecting more prompts addresses only the last. Second, we introduce a partial-smoothness condition suited to brittle safety behavior, since global Lipschitz continuity fails for models whose behavior flips under adversarial phrasing. Third, we make the terms estimable in practice, including an exact linear-program computation of the tightest bound of this form and, in the revision, a fixed-model validation with held-out prediction (Section 5.3).

W2 / Q1 (tightness with a fixed model). The supplementary experiment below fixes the target model and the underlying requests (Section 5.3, L449-503; Appendix G). The bound holds for every evaluation distribution. With the tighter irregularity term of Appendix B, its tightness is 12-29% on gated severity and 21-42% on compliance wherever it improves on the trivial bound, compared with 2.9-27.6% in the submitted cross-model analysis (Table 1). A held-out check confirms that constants fitted on half the requests predict a valid bound on the other half in every case (Table 12).

W3 / Q3 (comparison with conventional ASR). Our claim for severity-weighted ASR concerns what it estimates rather than how far it moves under distribution shift. Because risk is defined as expected severity, ASRsASR_s targets it directly, while flat ASR replaces severity with a binary indicator, which adds to the severity-validity term whenever unsafe outputs differ in severity (Section 4). For example, on qsuccessfulq_{\text{successful}} flat ASR saturates at 1.000 by construction while ASRsASR_s is 0.563 (Section 5.2, L424). Table 6 addresses a different question, how far each metric moves from its own p∗p* reference under distribution shift, and there the two behave similarly once their different scales are accounted for (mean relative gaps of 60% and 64%). We revised its caption and the corresponding text in Section 5.2 to make this explicit. More broadly, the framework's contribution to accuracy is diagnostic. Flat ASR on its own gives no indication of how far it may be from the risk under the threat model, whereas W1W_1, the smoothness profile, and judge reliability indicate when an estimate can be trusted. The fixed-model experiment shows this working: with smoothness fitted on half the requests, the bound gave a valid limit on the estimation error for the other half in every held-out case. The importance-weighted estimator (Section 4, Appendix C) is the component designed to correct distribution-shift bias directly, and evaluating it empirically under a realistic threat distribution is an important next step.

Q2 (sensitivity to the target distribution). We have not yet varied p∗p* in the new experiments, and we state this as a limitation in Section 5.3 (L497). Our results are defined relative to a stated p∗p* by design. The framework makes the dependence on the threat model explicit rather than removing it, and a different p∗p* asks a different question about the model. When a realistic threat distribution is hard to estimate, the framework suggests two practical options. First, a threat-model-specific approximation, even a synthetic one such as the user simulator of Section 5.1, can be stated and audited, which improves on an implicit uniform target (Section 3.2, Assumption 2). Second, when several candidate distributions are plausible, one can report the risk estimate and diagnostics under each, or take p∗p* to be a family of threat-model distributions and bound the worst case over it, which connects to distributionally robust optimization (Section 6). A systematic sensitivity analysis over alternative choices of p∗p* is a natural next step.

AuthorsOpenReview ↗

Additional experiment: fixed-model bound verification on prompt families

Motivation. Our hh-rlhf evidence used prompts answered by different models, so measured smoothness could reflect model differences. Here we fix one model and a set of underlying requests and vary only their phrasing (details in new PDF, Section 5.3 and Appendix G).

Setup. We took 50 harmful requests from WildJailbreak and built a 10-prompt family for each: the original, 3 paraphrases, 3 style rewrites (casual, formal, terse), and 3 of WildJailbreak's own jailbreak versions. Paraphrases and rewrites were generated with Qwen2.5-7B-Instruct, filtered automatically, and reviewed manually for meaning preservation. Mistral-7B-Instruct-v0.3 gave four responses per prompt (one greedy, three sampled), scored by Gemini with a StrongREJECT-style gated rubric (refusals score zero). A prompt's severity is its mean over responses, an estimate of expected severity, normalized so M=1M = 1; we also report compliance. The target p∗p* is uniform over the 350 non-jailbreak prompts. Evaluation sets mix in a growing share of jailbreak versions (0-100%), restrict to one prompt kind, or restrict to one of two topic clusters. We compute exact W1W_1 and the tightest bound of the Appendix B form by one linear program over LL and every LL-Lipschitz gg, against the trivial bound Rf(p∗)+Rf(q)R_f(p^*) + R_f(q).

ScoreJailbreak shareGapW1W_1BoundTrivialAlignment share
Gated severity10%0.0060.0440.0460.08853%
Gated severity30%0.0170.1330.0870.09942%
Compliance10%0.0270.0440.1270.40263%
Compliance50%0.1360.2220.3990.51154%

Findings.

  • The bound holds for every evaluation distribution and is nontrivial for moderate shifts, with tightness of 12-29% on gated severity and 21-42% on compliance (Figure 6).
  • Held-out. Fitting LL and gg on half the families and predicting on the other half without refitting, the gap stayed under the prediction in all 320 cases (20 splits), within 0.007 of the in-sample optimum (Table 12).
  • The M⋅δM \cdot \delta form is far looser, so the revision uses the tighter Appendix B form.
  • Irregularity concentrates in jailbreaks. Rewordings of a request differ in severity by 0.031 on average, against 0.092 between a rewording and a jailbreak version.
  • Response randomness matters. Using only the greedy response narrows where the bound is nontrivial (Table 14).
  • Topic shift. Topic clusters sit at W1W_1 of 0.28-0.36 yet change risk by at most 0.004, so large W1W_1 need not mean large error.

Limitations. One model, low base severity, mostly synthetic prompts, a fixed p∗p*, and informative bounds only for shifts toward related prompts.

Reviewer_YzBFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper approaches red-teaming success metrics from a theoretical perspective and argues that existing standards like ASR are unreliable. The authors instead propose a theoretical, probabilistic model that aims to capture the underlying safety risk of a model in a deployment condition (i.e. under a particular prompt distribution and threat model). The framework relies on a decomposition of estimation error between the risk measured during a red-teaming evaluation and the actual risk under the target distribution. The contributing error terms are alignment (lower if safety-audit prompts are more similar to the deployment distribution), irregularity (lower if the model behaves "smoothly"), severity validity (error due to, e.g., biased LLM-judges), and sampling error (error due to estimating with a finite number of prompts). The authors also present two empirical analyses supporting aspects of their theoretical framework, demonstrating the effect of evaluation distributions on measured ASR and estimating differences in severity smoothness across models.

Critical strengths
  • The paper takes a systematic look at safety metrics and approaches the measurement of risk from a systematic point of view and makes a good case that the field should try to estimate "model safety risk" as defined in the paper, rather than focusing on ASR on a particular prompt dataset. The authors clearly articulate gaps between what the current practice is and what it could be, and I appreciate the attempt to bring more rigor into the red-teaming field which often suffers from unstable baselines/metrics and results which are difficult to compare.
Critical weaknesses
  • The empirical evidence of the usefulness of the framework is rather weak at the current stage as the experiments were done on few models and limited datasets.
  • The theoretical bounds are not meaningful in practice (Table 3)
Questions for authors

W_1 is computed with a particular embedding model. How does the choice of embedding model affect the validity of the bounds? What properties would an ideal embedder have to have for the purposes of this paper?

Is there a clear path towards tightening the bounds towards more practical regimes?

Since each hh-rlhf record reflects a single sampled response, r_i measures the smoothness of a noisy realization rather than of expected severity. Could the lower sensitivity of preference-trained models in Table 2 be caused by lower response variance (e.g., more consistent refusals) rather than a smoother expected-severity surface, and is there a way to separate the two?

L1459: the linear trend reports perfect correlation rho=1 when this is clearly not the case. A few sections are a bit tough/LLM-y (L399, L426)

2 comments

We thank the reviewer for the constructive questions. We have revised the paper to address each point; line numbers refer to the revised PDF.

Limited experiments. We added a fixed-model experiment on 50 families of matched prompts, with four responses per prompt and a held-out check (Section 5.3, Appendix G; summarized below). It uses a single target model, which we state as a limitation (L497).

Bounds not meaningful in practice / path to tightening. The looseness of the submitted bounds came mainly from the M⋅δM \cdot \delta irregularity term, which charges every irregular prompt the full severity range. Using the tighter form in Appendix B (Equation 15), computed exactly by a linear program over LL and every LL-Lipschitz gg, the fixed-model bound is 2-8 times the gap and well below the trivial bound Rf(p∗)+Rf(q)R_f(p*) + R_f(q) for moderate shifts, with the alignment term carrying 42-64% of it (Table 1). Three levers tighten the bound in practice. The first is the tighter irregularity term. The second is averaging several responses per prompt, which widened the range where the bound improves on the trivial one from 30% to about 50% jailbreak share on gated severity and from 40% to 90% on compliance (Table 14). The third is evaluation designs in which the shift between qq and p∗p^* moves part of the probability mass to nearby prompts rather than all of it to unrelated ones (L497).

Choice of embedding. The bound is valid for any ground metric and invariant to its scale (Section 2, L206), since (L,δ)(L, \delta) are defined relative to it, but its tightness depends on whether the metric places prompts that elicit similar harm close together, and the metric must be chosen without reference to severity labels to avoid circularity. The fixed-model experiment shows the cost of a topic-sensitive embedder. Two topic clusters sit at W1W_1 of 0.28 and 0.36 from the target yet change risk by at most 0.004, so the bound is valid but conservative there (Section 5.3, L497). A second property matters for finite samples. In the full 768-dimensional embedding, the public benchmarks of Section 5.1 sit only 2.1-2.4 times farther from p∗p* than a second sample of p∗p* itself, while after projecting to two principal components fit on the prompts alone they sit 3.7-9.8 times farther, with the same ordering at every dimension (new Appendix F.4, Table 10). An ideal embedder for this purpose is therefore fixed independently of severity, captures the harm-relevant content of a prompt rather than its topic, and is low-dimensional enough that sampling noise does not dominate W1W_1.

Noisy realization vs. expected severity. We agree, and the new experiment separates the two for a fixed model. The standard deviation of severity across a prompt's four responses is 0.026-0.043, comparable to mean severity, and using only the greedy response instead of the four-response mean substantially narrows the range where the bound benefits from smoothness (Section 5.3, L486; Table 14). Part of the apparent irregularity of single responses is therefore sampling randomness. For hh-rlhf, which has one response per record, the two cannot be separated. We revised Section 5.2 (L409-419) to present the per-stratum differences (now Table 3) as descriptive of the corpus, naming consistent refusal as one possible cause alongside differences in prompts and transcript-level severity.

Minor. The value in the legend at L1459 is Spearman's rank correlation, which is 1.00 because the four distributions have the same ordering under W1W_1 and under the ASR gap; it does not describe the fit of the plotted line (Pearson r=0.81r = 0.81). We agree the legend invites this reading, so the revised caption (Figure 1, L1524-1530) now states this explicitly and notes that with only four points neither statistic is informative. We also rewrote the passages at L399 and L426 of the submitted version.

AuthorsOpenReview ↗

Additional experiment: fixed-model bound verification on prompt families

Motivation. Our hh-rlhf evidence used prompts answered by different models, so measured smoothness could reflect model differences. Here we fix one model and a set of underlying requests and vary only their phrasing (details in new PDF, Section 5.3 and Appendix G).

Setup. We took 50 harmful requests from WildJailbreak and built a 10-prompt family for each: the original, 3 paraphrases, 3 style rewrites (casual, formal, terse), and 3 of WildJailbreak's own jailbreak versions. Paraphrases and rewrites were generated with Qwen2.5-7B-Instruct, filtered automatically, and reviewed manually for meaning preservation. Mistral-7B-Instruct-v0.3 gave four responses per prompt (one greedy, three sampled), scored by Gemini with a StrongREJECT-style gated rubric (refusals score zero). A prompt's severity is its mean over responses, an estimate of expected severity, normalized so M=1M = 1; we also report compliance. The target p∗p* is uniform over the 350 non-jailbreak prompts. Evaluation sets mix in a growing share of jailbreak versions (0-100%), restrict to one prompt kind, or restrict to one of two topic clusters. We compute exact W1W_1 and the tightest bound of the Appendix B form by one linear program over LL and every LL-Lipschitz gg, against the trivial bound Rf(p∗)+Rf(q)R_f(p^*) + R_f(q).

ScoreJailbreak shareGapW1W_1BoundTrivialAlignment share
Gated severity10%0.0060.0440.0460.08853%
Gated severity30%0.0170.1330.0870.09942%
Compliance10%0.0270.0440.1270.40263%
Compliance50%0.1360.2220.3990.51154%

Findings.

  • The bound holds for every evaluation distribution and is nontrivial for moderate shifts, with tightness of 12-29% on gated severity and 21-42% on compliance (Figure 6).
  • Held-out. Fitting LL and gg on half the families and predicting on the other half without refitting, the gap stayed under the prediction in all 320 cases (20 splits), within 0.007 of the in-sample optimum (Table 12).
  • The M⋅δM \cdot \delta form is far looser, so the revision uses the tighter Appendix B form.
  • Irregularity concentrates in jailbreaks. Rewordings of a request differ in severity by 0.031 on average, against 0.092 between a rewording and a jailbreak version.
  • Response randomness matters. Using only the greedy response narrows where the bound is nontrivial (Table 14).
  • Topic shift. Topic clusters sit at W1W_1 of 0.28-0.36 yet change risk by at most 0.004, so large W1W_1 need not mean large error.

Limitations. One model, low base severity, mostly synthetic prompts, a fixed p∗p*, and informative bounds only for shifts toward related prompts.

Reviewer_LUtpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats red-teaming as a measurement problem and defines the target quantity as expected response severity under a specified threat-model distribution. It bounds the difference between this quantity and an empirical estimate using distributional mismatch, partial smoothness of severity, judge bias, and sampling error. The proposed partial-smoothness condition allows a small irregular region and yields a Wasserstein-based bound. The paper then recommends coverage, smoothness, severity-weighting, importance-weighting, and judge-reliability diagnostics. Experiments illustrate variation in attack success across prompt distributions for a fixed model and judge, and estimate local severity sensitivity on an existing human red-teaming corpus stratified by model type and size.

Critical strengths
  1. The paper makes the target of safety measurement explicit. Separating the target distribution, evaluation distribution, severity assessment, and finite-sample error is a useful way to prevent overinterpretation of ASR. Corollary 3 clearly explains why collecting more prompts from the same distribution cannot remove systematic mismatch or judge bias. The discussion also correctly distinguishes judge agreement from absence of shared bias.

  2. Partial smoothness offers an interpretable extension of a standard transport bound. Allowing an irregular set is more appropriate for brittle safety behavior than assuming global Lipschitz continuity without qualification. Appendix B gives a transparent extension-and-decomposition proof, while Section 5.2 correctly recognizes that nearest-neighbor checks alone do not certify the all-pairs condition. The explicit certification procedure and disclosure of empirical confounds are valuable aspects of the analysis.

Critical weaknesses
  1. Several central recommendations treat a sufficient upper bound as if it supplied necessary conditions or a ranking principle. Section 3.1 says estimates are informative “only when” the distributions are close and the severity function is regular, and Section 6 proposes reducing Wasserstein distance per unit effort as a definition of red-teaming efficacy. Equation 4 does not establish either claim. For example, on prompts {0,1,2} with ordinary distance, let the target be a point mass at 0 and severity be (0,1,0). Severity is globally 1-Lipschitz. An evaluation distribution concentrated at 1 has distance 1 and risk gap 1, whereas one concentrated at 2 has distance 2 and gap 0. Thus larger distance need not mean worse risk estimation, even with fixed smoothness. Appendix F.1's description of the observed ordering as predicted by the bound is therefore too strong. The framework also needs to preserve the distinction between estimating a fixed risk and discovering rare failures, for which concentrating on successful attacks can be appropriate.

  2. The empirical study does not yet validate the proposed smoothness diagnostics as measures of fixed-model risk generalization. The fixed-model experiment has only four comparison distributions and does not estimate the smoothness terms. The larger corpus supplies those terms, but, as the authors explicitly acknowledge, its reference and evaluation records come from different models and cannot instantiate Equation 4 for one fixed model. Stratifying by model type and size helps but does not hold the prompt distribution, neighborhood density, or conversation continuation fixed. Moreover, severity is the minimum harmlessness score across a transcript, while distances are computed from prompts; divergent conversations can produce different severities for identical openings. Lower observed sensitivity for preference-trained strata therefore does not by itself identify smoother expected safety behavior under matched prompts. The principal empirical novelty needs a controlled fixed-model or matched-prompt validation, rather than only corpus-level descriptive statistics.

Questions for authors
  1. Theoretical clarification: Which recommendations require only a sufficient bound, and which require an additional relationship between Wasserstein distance and actual estimation error? Can the “only when,” efficacy-definition, and predicted-ordering claims be qualified accordingly?
  2. Measurement clarification: Can the smoothness comparison be made on matched prompts and comparable neighborhood densities across model strata, accounting for multiple possible continuations and the transcript-level severity score? How much of the reported difference remains after these controls?
  3. Validation clarification: Is there evidence that smoothness and alignment diagnostics estimated on one sample predict risk-estimation error on a separate sample for a fixed model? The exact finite-corpus certification is useful, but it does not by itself establish generalization to unseen prompts.

These questions address the paper's present claims about valid measurement; they do not require a new red-teaming attack method.

2 comments

We thank the reviewer for the careful reading and the precise counterexample. We have revised the paper to address each point; line numbers refer to the revised PDF.

W1 / Q1 (sufficient vs. necessary). We agree that Equation 4 is a sufficient condition and does not imply that a larger W1W_1 means a larger error. We made three changes. (i) Section 3.1 (L234-237) now says the bound guarantees that R^f(q)\hat{R}_f(q) is informative about Rf(p∗)R_f(p*) when alignment, regularity, and severity validity hold, and states explicitly that the guarantee is sufficient rather than necessary, so a larger W1(q,p∗)W_1(q,p^*) permits but does not imply a larger error. (ii) In Section 6 (L517-524), the efficacy proposal is now a heuristic for exercises that aim to estimate risk, which we distinguish from exercises that aim to discover rare failures, where concentrating on successful attacks is appropriate. The recommendation for automated red-teaming is qualified in the same way. (iii) Section 5.1 (L380) and Appendix F.1 (L1574) now describe the observed ordering as consistent with the bound rather than predicted by it, and note that the bound implies no ordering.

On which recommendations rest on the bound alone: recommendations stated in terms of the guarantee, such as that a corpus closer to p∗p* yields a tighter bound and that W1W_1 and the smoothness profile should be reported as diagnostics (Sections 3.1 and 4), require only the sufficient bound. Claims about the ordering of actual errors would require an additional relationship between distance and error, and we have qualified the three such claims the reviewer identified. The new fixed-model experiment also gives an empirical instance of the reviewer's point (Section 5.3, L497). Restricting qq to one topic cluster places it at W1W_1 of 0.28-0.36 from p∗p^* while changing risk by at most 0.004.

W2 / Q2 (fixed-model, matched-prompt validation). We added an experiment that holds one target model and a set of underlying requests fixed and varies only their phrasing (Section 5.3, L449-503; Appendix G; summarized below). It uses single-turn prompts, so there are no divergent continuations, and it averages four sampled responses per prompt to estimate expected severity rather than a single realization. With these controls the bound holds for every evaluation distribution and is nontrivial for moderate shifts, and the irregularity concentrates in adversarial versions of the same request. We have not repeated the training-paradigm comparison under matched prompts across model strata. We therefore revised Section 5.2 (L409-419) to present it as a description of the corpus rather than as evidence that preference training smooths expected safety behavior. The paragraph now names the confounds the reviewer identifies (different prompts and prompt density across strata, single sampled conversations, and transcript-level severity against prompt-only distances), and notes that Section 5.3 shows, for one model, that response variability alone changes how much smoothness the bound can use. We removed the corresponding claims from the abstract, introduction, and Appendix E.

Q3 (held-out validation). Yes, for a fixed model. Over 20 random splits of the 50 prompt families into halves, we fit L⋆L^\star and the smooth function gg on one half and predict the bound on the other half without refitting. The held-out gap stayed below the prediction in all 320 held-out cases (16 settings over 20 splits), and predictions were within 0.007 of the bound optimized directly on the held-out half (Section 5.3, L480; Appendix G.5, Table 12). This establishes transfer to unseen requests from the same source, not to arbitrary prompt sources, which we state in Section 5.3 (L497).

AuthorsOpenReview ↗

Additional experiment: fixed-model bound verification on matched prompt families

Motivation. Our hh-rlhf evidence used prompts answered by different models, so measured smoothness could reflect model differences. Here we fix one model and a set of underlying requests and vary only their phrasing (details in new PDF, Section 5.3 and Appendix G).

Setup. We took 50 harmful requests from WildJailbreak and built a 10-prompt family for each: the original, 3 paraphrases, 3 style rewrites (casual, formal, terse), and 3 of WildJailbreak's own jailbreak versions. Paraphrases and rewrites were generated with Qwen2.5-7B-Instruct, filtered automatically, and reviewed manually for meaning preservation. Mistral-7B-Instruct-v0.3 gave four responses per prompt (one greedy, three sampled), scored by Gemini with a StrongREJECT-style gated rubric (refusals score zero). A prompt's severity is its mean over responses, an estimate of expected severity, normalized so M=1M = 1; we also report compliance. The target p∗p* is uniform over the 350 non-jailbreak prompts. Evaluation sets mix in a growing share of jailbreak versions (0-100%), restrict to one prompt kind, or restrict to one of two topic clusters. We compute exact W1W_1 and the tightest bound of the Appendix B form by one linear program over LL and every LL-Lipschitz gg, against the trivial bound Rf(p∗)+Rf(q)R_f(p^*) + R_f(q).

ScoreJailbreak shareGapW1W_1BoundTrivialAlignment share
Gated severity10%0.0060.0440.0460.08853%
Gated severity30%0.0170.1330.0870.09942%
Compliance10%0.0270.0440.1270.40263%
Compliance50%0.1360.2220.3990.51154%

Findings.

  • The bound holds for every evaluation distribution and is nontrivial for moderate shifts, with tightness of 12-29% on gated severity and 21-42% on compliance (Figure 6).
  • Held-out. Fitting LL and gg on half the families and predicting on the other half without refitting, the gap stayed under the prediction in all 320 cases (20 splits), within 0.007 of the in-sample optimum (Table 12).
  • The M⋅δM \cdot \delta form is far looser, so the revision uses the tighter Appendix B form.
  • Irregularity concentrates in jailbreaks. Rewordings of a request differ in severity by 0.031 on average, against 0.092 between a rewording and a jailbreak version.
  • Response randomness matters. Using only the greedy response narrows where the bound is nontrivial (Table 14).
  • Topic shift. Topic clusters sit at W1W_1 of 0.28-0.36 yet change risk by at most 0.004, so large W1W_1 need not mean large error.

Limitations. One model, low base severity, mostly synthetic prompts, a fixed p∗p*, and informative bounds only for shifts toward related prompts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.