acceptodds
Under review as a conference paper at ICLR 2027

How Jailbreak Attacks Inform LLM Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions

Abstract

Jailbreak attacks on large language models are usually evaluated by attacker-centric metrics such as attack success rate (ASR), yet an attack that breaks a model is not necessarily useful for improving its safety. We propose a defender-centric view of jailbreak evaluation, where attacks are evaluated by the downstream safety improvements they enable when used as red-teaming data for safety training. Building on this view, we introduce (), a setting-agnostic framework for attributing and selecting jailbreak attacks from black-box subset utility observations. A-MESS estimates , a Shapley-based score that attributes marginal utility to individual attacks and selects compact attack subsets under user-specified budgets via greedy or surrogate-based optimization. Across controlled utility landscapes and real LLM safety settings, we find that ASR rankings are weakly aligned with defender-centric utility, that AttackSHAP can be estimated accurately with limited utility queries, and that directly optimizing subsets yields stronger safety utility than attacker-centric or attribution-only selection. These results suggest evaluating jailbreak attacks as resources for improving safety, not only as tools for breaking models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm safety, jailbreak, shapley, red-teaming

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xpvhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper reframes jailbreak attack evaluation from an attacker-centric view (measured by attack success rate, ASR) to a defender-centric view, where attacks are valued by the downstream safety improvement they enable when used as red-teaming data for safety training. The authors formalize each safety setting θ as a black-box utility function v_θ(S) over attack subsets, and introduce:

  1. AttackSHAP — a Shapley-based attribution score treating attacks as players and attack subsets as coalitions, quantifying each attack's average marginal contribution to downstream defense utility (Eq. 1).
  2. A-MESS (Minimal Effective Attack-Subset Selection) — a setting-agnostic framework with two solvers: A-MESS-GREEDY (queries true marginal utility for single-budget selection) and A-MESS-SURROGATE (learns a surrogate utility landscape from sampled subset evaluations, enabling AttackSHAP estimation and multi-budget subset search), plus A-MESS-ACQUISITION, an active query strategy that balances coalition-size coverage with within-size model disagreement (query-by-committee). Experiments on 90 controlled synthetic utility landscapes (n=20, exact 2²⁰ enumeration) and three real LLM safety settings (Qwen2.5-7B + SFT; Ministral-3-8B + ICD/SFT, n=10, all 1,024 subsets evaluated) show that (i) ASR rankings align only weakly with defender utility (Spearman ρ = 0.188, Kendall τ_b = 0.111), (ii) A-MESS-ACQUISITION accelerates early Shapley error reduction and lowers subset-selection gaps vs. passive acquisition, and (iii) direct subset optimization (A-MESS) recovers more oracle gain (≥82.69% NG_k greedy; 84.52% surrogate at k=9) than attribution-only top-k selection (60–71%), with the surrogate achieving utility 0.8542 vs. 0.6606 for the strongest baseline at k=4 in the real setting.
Critical strengths
  • S1 — Meaningful problem reframing. The shift from "which attacks break models" to "which attacks improve defended models" is well motivated (Intro L042–L053) and directly addresses a gap: prior defenses show attack data helps safety but rarely ask which attacks are most useful.
  • S2 — Complete and principled method design. Attribution (AttackSHAP), budgeted subset selection (Greedy/Surrogate), and active acquisition (Eq. 3, Algorithm 1) form a logically coherent pipeline; the black-box, setting-agnostic design fits proprietary real-world safety pipelines.
  • S3 — Strong experimental coverage. 90 synthetic landscapes with exact oracle AttackSHAP references plus 3 real LLM settings with all 2¹⁰ = 1,024 subsets enumerated; surrogate prediction on real utilities reaches R² = 0.952 (MAE 0.037) with only 400 training labels (Table 3).
  • S4 — Clear empirical separation of attribution vs. selection. Table 4 shows attribution-based top-k plateaus at ~60–71% Normalized Gain while direct subset optimization reaches ≥82.69%, cleanly demonstrating that individual importance ranking is insufficient for subset-level decisions.
  • S5 — Honesty and transparency. The authors proactively disclose the single-evaluation limitation (App. G), the train-eval overlap (App. D), cost non-alignment (Sec. 4.5), and provide utility-weight sensitivity (App. J, Tables 12–13).
Critical weaknesses
  • W1 — The core negative-correlation claim is statistically fragile. Spearman ρ = 0.188 and Kendall τ_b = 0.111 are computed from n = 10 attack categories with a single greedy evaluation per defense subset; the confidence interval at this sample size is extremely wide, and Appendix G (L884–L886) explicitly concedes no repeated-generation significance is established. The paper's central thesis therefore rests on weak evidence.
  • W2 — Training–evaluation overlap contaminates the utility signal. The JBB evaluation includes the defense behaviors used for SFT/ICD construction (Appendix D, L805–L807); measured v_θ(S) may reflect overfitting to the training attacks rather than generalizable safety improvement.
  • W3 — Defender utility is a single hand-weighted scalar. Eq. (10)'s weights (0.9 ASR / 0.05 over-refusal / 0.05 capability) are arbitrary; with such small non-ASR weights, the "defender-centric" utility is nearly a rescaled ASR reduction, which risks overstating the divergence from attacker-centric metrics. Sensitivity analysis is limited to two alternative weightings in Appendix J.
  • W4 — Query budgets are not matched across methods. Table 5 compares A-MESS solvers against baselines with different utility-evaluation budgets (e.g., greedy uses nk − k(k−1)/2 true queries vs. M for the surrogate); the authors acknowledge this "illustrates attainable utility rather than a matched-cost advantage" (L427–L429), leaving the practical efficiency claim unverified.
  • W5 — Limited scalability and unvalidated surrogate assumptions. Experiments use n = 10 (real) and n = 20 (synthetic); how MLP-based membership-vector surrogates and Shapley estimation scale to dozens or hundreds of attack categories with more complex interactions is neither discussed nor empirically probed, and the mechanism gap between synthetic landscapes and real defense pipelines (LoRA training stochasticity, data shuffling) is not validated.
Questions for authors
  1. (Re: W1) If each subset utility were estimated from 5–10 repeated generation evaluations and the correlations recomputed, would Spearman ρ / Kendall τ_b still be significantly distinguishable from zero at n = 10, or would the ranking-misalignment conclusion require more attack categories?
  2. (Re: W2) Can you provide a strictly held-out evaluation (e.g., excluding the defense behaviors from the JBB benchmark, or evaluating on a disjoint attack set) to show that selected high-utility subsets generalize rather than overfit the defense-training data?
  3. (Re: W4/W5) How do the relative rankings of A-MESS-GREEDY vs. A-MESS-SURROGATE vs. TOPK-ϕ̂ change under a matched true-utility-query budget, and how does surrogate Shapley estimation error degrade as n grows to 30–50 attack categories?
Reviewer_wWLuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to evaluate jailbreak methods by the downstream defense utility of their generated data. the paper proposes A-MESS, which combines Shapley attribution, greedy subset selection, and a learned utility surrogate with size-balanced, disagreement-based methods. Evaluation covers 90 synthetic landscapes and 3 model/defense settings with 10 attacks. Results show weak agreement between attack-success rankings and singleton defense utility, and stronger selected-subset utility from direct optimization than from individual rankings under the tested protocols.

Critical strengths
  1. the idea of separating attack success from usefulness for defense addresses a practical red-teaming resource-allocation problem, the problem itself is meaningful.
  2. both synthetic and real-world evaluations support the idea, which is great
Critical weaknesses
  1. The surrogate uses 500 utility evaluations and the displayed run is selected from five seeds, whereas greedy needs only 34 queries for 4 attacks. The paper acknowledges unequal costs, but this comparison cannot substantiate a practical efficiency advantage (Section 4.5; Appendix I).
  2. Subset search and reporting use the same cached evaluation protocol. SFT uses only three behaviors, which also occur in the JBB evaluation, anyway to prove the generalization of the method on held-out evaluations?
Questions for authors

n/a

Reviewer_VxZZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates jailbreak attacks from the defender's perspective, asking how useful their data are for improving a defense rather than considering attack success alone. It defines a cooperative game using the utility of defenses constructed from attack subsets, introduces AttackSHAP to measure individual attacks' average marginal contributions, and uses greedy and surrogate-based A-MESS solvers to select attack combinations directly. The surrogate's acquisition strategy combines coverage across subset sizes with model disagreement to reduce expensive utility queries.

The evaluation includes 90 synthetic utility landscapes and three real defense settings across two models. Each real setting evaluates all 1,024 subsets of ten attacks. Under the chosen utility definitions, attack success and defensive contribution are weakly aligned, and direct subset optimization can outperform selection based on individual attribution rankings in some settings.

Critical strengths

1. Evaluating attacks by their contribution to defense improvement addresses a meaningful practical question.

Safety research should assess not only how effectively an attack breaks a model, but also how useful its data are for improving defenses. The paper provides empirical examples where these objectives differ and distinguishes an individual attack's average contribution from the value of an attack combination. This perspective is directly relevant to selecting data for safety training or in-context defenses.

2. Exhaustive subset evaluation provides a useful basis for analyzing interactions and testing selection methods.

The real settings enumerate all 1,024 subsets of ten attacks, while the synthetic settings provide computable attribution and optimal-subset references. The experiments show that direct subset optimization can outperform selection based only on individual attribution, and the synthetic acquisition comparisons provide initial support for the proposed strategy. These designs give the evaluation clear reference points rather than relying on a few hand-selected combinations.

Critical weaknesses

1. The real experiments do not convincingly establish the trade-off between query cost and benefit, or the stability of the main selection results.

Appendix I uses 500 real utility queries for the surrogate, whereas the reported subsets of sizes two and four have only 45 and 210 candidates. For these specific selection tasks, exhaustive evaluation of the relevant candidates would cost less. The surrogate could potentially amortize its cost across multiple subset budgets or attribution tasks, but the real results do not sufficiently demonstrate that benefit or establish a reliable advantage under matched query budgets.

The main text also states that the displayed run was selected from five model seeds, without adequately specifying the selection criterion or reporting all seeds' final selection outcomes. The authors acknowledge that query costs differ across methods, which is helpful, but the evidence still does not show whether the observed advantage is consistently reproducible. The selection procedure needs clarification; the available information does not establish that the run was chosen by its test performance.

2. The composite utility and post-selection evaluation do not sufficiently establish the specific safety gains and their costs.

The main utility assigns 90% weight to relative attack success reduction and clips that contribution to be nonnegative. An increase in an attack's success rate therefore receives the same zero contribution as no change, and deterioration can be masked by improvements elsewhere. The main subset-selection results do not sufficiently report absolute attack success rates, over-refusal, and general-capability accuracy. Alternative-weight experiments are provided, but they do not replace reporting these components separately.

Independent post-selection evaluation also needs strengthening. The appendix discloses that the full JBB evaluation includes three behaviors used for defense training. This limited overlap should not be described as leakage across the entire evaluation, but it should be excluded and checked separately. More generally, holding out utility labels for attack subsets is not equivalent to holding out the evaluation questions used for selection and final assessment. These limitations weaken the interpretation of higher utility as a stable safety improvement.

Questions for authors

1. Query efficiency and seed stability: clarification and analysis of existing results.

What criterion was used to select the displayed surrogate run from the five model seeds? Could you report the selected subset and its utility for every seed? Using the existing exhaustive utility tables, could you compare active acquisition, sampling balanced across subset sizes, and relevant selection methods under matched query budgets below the corresponding exhaustive-search cost? These results would directly affect my assessment of practical efficiency and stability.

2. Specific safety gains: disaggregated reporting and limited independent validation.

Could you report absolute attack success rates, over-refusal rates, and capability accuracy for the final attack combinations and main baselines? Could you separate the questions used for subset selection from those used for final evaluation, exclude defense-training behaviors from the latter, and check whether the main advantages remain? The request concerns the benefits and costs in the reported settings, not universal generalization to all unseen attacks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.