acceptodds
Withdrawn from ICLR 2027

LLM Safety: What Are Red Teams and Blue Teams Really Competing For?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) are trained to refuse harmful requests, yet carefully crafted jailbreak prompts continue to bypass these safeguards. The standard metric for evaluating attack effectiveness is the Attack Success Rate (ASR). However, we demonstrate that ASR is systematically misleading: it primarily reflects the agreement between the supervisor used to guide prompt generation and the evaluator used to judge success, rather than the intrinsic harmfulness of the model's output. This supervisor-evaluator gap inflates reported success rates, as many attacks that score positively simply exploit shared biases between the two classifiers without eliciting genuinely harmful content. Compounding this issue, the common practice of optimizing for refusal avoidance induces semantic drift, where a malicious query such as “how to make a bomb” is mutated into a benign variant like “how to make a bomb chicken” that evades refusal but produces harmless output. We argue that jailbreak research should shift its objective from making the model say yes to making the model produce actually harmful content. This reframes the problem as an arms race: the red team must learn the blue team's safety boundary to identify blind spots, while the blue team must expand its coverage to close them. We formalize this dynamic through the lens of harm discriminators, showing that attack success depends on the red team's ability to construct a discriminator with high recall relative to false positive rate, and to approximate the blue team's discriminator in black-box settings. Our experiments demonstrate that when the red team directly leverages the blue team's own evaluator as its supervision signal, it discovers truly effective jailbreaks substantially faster. This work offers a fresh perspective on LLM safety testing: the decisive factor is not the sophistication of attack algorithms, but the ability to learn the opponent's hidden decision boundary first.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
jailbreak, attack success rate (ASR), supervisor-evaluator mismatch

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ivNpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing jailbreak attacks optimize for refusal avoidance rather than harmful output. This causes semantic drift: a query like “how to make a bomb” mutates into a benign variant such as “how to make a bomb chicken,” which avoids refusal but yields harmless content. The authors further argue that reported ASR mainly reflects agreement between the supervisor used during search and the evaluator used for scoring, rather than whether outputs are actually harmful.

Critical strengths

Strength 1: Clear diagnosis of semantic drift. The paper shows concretely, through the “bomb → bomb chicken” examples and the decomposition in Eq. 3, that refusal-avoidance objectives can evade safeguards while producing harmless output. Why it matters: This is a real weakness of keyword-based search signals, and the framing is useful. However, it overlaps substantially with PAIR, TAP, and StrongREJECT, which limits its weight.

Strength 2: Consistent empirical gains. Swapping in a harmfulness discriminator improves GCG, AutoDAN, and BEAST on all 12 models, raising average ASR from roughly 63–72% to about 89.5%. Why it matters: This is the paper’s strongest evidence that the search signal matters more than the search algorithm. Its weight is reduced by the shared reliance on human annotation for both the supervisor and the evaluator, and by the missing comparison to existing judge-guided attacks.

Critical weaknesses

Weakness: None of the three main contributions is new.

  1. Refusal avoidance ≠ harmful output (semantic drift). That non-refusal does not imply harmful content is well known. StrongREJECT (Souly et al., 2024) explicitly studies such "empty jailbreaks," and HarmBench (Mazeika et al., 2024) adopted a harmfulness classifier instead of refusal keywords for the same reason.
  2. Harmfulness-guided attack search. Using a judge's harmfulness score to guide jailbreak search is already standard practice in PAIR (Chao et al., 2023) and TAP (Mehrotra et al., 2023). Neither is compared against experimentally.
  3. Red/blue arms race with a surrogate discriminator. Training a surrogate model through queries to attack a black-box defender is the classic substitute-model attack (Papernot et al., 2017). Iterative red/blue co-evolution for LLM safety is also explored in MART (Ge et al., 2024).

Why it matters: With all three ideas already established, the paper's contribution reduces to applying known techniques to older refusal-based attacks. Without comparisons to these works, that is not a sufficient advance for acceptance.

Questions for authors

I would like to ask the author to explain more about the novelty of the claim in this paper.

Reviewer_s8oBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the existing LLM red-teaming is limited by the refusal-avoiding optimization and disagreement between the attack supervisor and evaluator. It further reframes jailbreaking as learning a good discriminator and proposes a new harm elicitation objective utilizing such discriminator to optimize jailbreak prompts. Experiments on three attack algorithms against various LLMs demonstrate substantial improvements in ASR with the proposed harm elicitation objective.

Critical strengths
  • The paper provides useful insights that refusal avoidance should be separated from actual harmful generation. The proposed discriminator-based framework also provides a new lens for understanding the dynamics between the attacker and defender.
  • The paper evaluates the proposed harm elicitation objective on different attack methods and various target LLMs to highlight its generalizability, and further validates the proposed discriminator-based framework with experiments simulating multi-round attack-defense scenarios.
Critical weaknesses
  • The novelty of the harm elicitation objective requires further justification. PAIR (Chao et al., 2023) and TAP (Mehrotra et al., 2023), as both cited by the paper, already advance beyond refusal avoidance to utilize the judge’s feedback on the victim’s generated responses to guide attack search. The authors should compare their proposed objective with these methods and explain what distinguishes their method from existing harmfulness-guided optimization.
  • The paper’s claim on supervisor-evaluator gap that “many attacks that score positively simply exploit shared biases between the two classifiers without eliciting genuinely harmful content”requires further support with empirical evidence. The authors could strengthen this claim by quantifying how often supervisor and evaluator both judge an attack successful while the generated content is not harmful for existing attack methods.
Questions for authors
  • Can the proposed harm elicitation objective be applied to jailbreak black-box models with limited query budget? Since the proposed harm elicitation objective in equation 7 requires responses from the target model, please clarify the query cost of optimization and how ASR varies with the available query budget.
  • Clarification required: for equation 9, should “max” be replaced with “argmax”? The equation appears to show two objectives share the same maximizers, rather than their maximum values are equal.
Reviewer_MJBTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether the Attack Success Rate (ASR) metric used for jailbreak attacks truly reflects attack harmfulness, and finds the harm elicitation gap: ASR mainly reflects the agreement between the "supervisor that guides the search" and the "evaluator that judges success," and optimizing to evade refusal causes semantic drift (e.g., mutating "how to make a bomb" into "how to make a bomb chicken"). The paper proposes an attack framework with harm elicitation as the objective, defines the Conditional Harm Probability (CHP) and the effective attack surface (the region covered by the red-team discriminator but not by the blue team), proves that CHP is determined by the ratio of the red-team discriminator's recall to its false positive rate (Lemma 1), and conducts experiments with GCG, AutoDAN, and BEAST on 10+ models plus a four-round adversarial experiment with alternating red/blue discriminator updates, demonstrating that switching the objective from refusal-avoidance to harm elicitation raises the average ASR from 63.10 / 72.20 / 70.39 to 89.61 / 89.53 / 89.47, and that the surrogate enables the red team to recover CHP from 26% to 71% in the fourth round.

Critical strengths
  1. The problem is accurately located and the formalization is clear: The view of decomposing ASR into a supervisor-evaluator conditional agreement rate is novel and convincing. The harm elicitation objective proposed in the main text clearly distinguishes the two different goals of "bypassing refusal" and "producing harmful content," laying the correct foundation for the subsequent analysis.
  2. The examples in Table 1 are intuitive and powerful: Five variants of the same malicious question "how to make a bomb" (bomb chicken, bomb pop, bombastic party, bomb shelter, bomb squad) and the corresponding model responses (recipe, ice pop, party, shelter, emergency squad) vividly demonstrate the ubiquity of semantic drift.
  3. The experiments cover a wide range of models and algorithms: Table 2 compares Original vs Ours for three algorithms on 10+ models, with consistent average ASR improvements. Weaker models such as Llama 2 7B show smaller gains (GCG 30.00→31.00), indicating the method depends on model capability, but the overall trend is robust.
Critical weaknesses
  1. The proof of Lemma 1 contains a logical leap: Lemma 1 claims that the maximum CHP equals the maximum of "log recall minus log false positive rate," but the derivation in Appendix A only shows that the two optima coincide when the prior ratio does not vary with the red-team discriminator. CHP is a probability between 0 and 1, while that log difference can take any real value; the proof does not establish the equality of the two maxima.
  2. The Abstract and Section 1 claim that "when different methods are compared with the same supervisor–evaluator pair, performance differences are usually not statistically significant," but the main text contains no statistical test: The closest observation is that the three Ours rows in Table 2 have almost identical average ASR (89.61 / 89.53 / 89.47), but the authors perform no significance test and report no confidence intervals.
  3. The claims of "substantially faster" / "higher query efficiency" lack direct evidence: The Abstract says that when the red team directly uses the blue-team evaluator for supervision, "effective jailbreaks are found much faster." However, the experiment budget is at most 300 queries per model for all methods, and the table reports ASR under that budget, not the number of queries needed to reach a given ASR. There is no query-count–success-rate curve.
Questions for authors
  1. Should the equality in Lemma 1 be an inequality? The proof only establishes the equivalence of the optima, not the equality of the two maxima. Can Lemma 1's statement be corrected, or can a complete proof be provided?
  2. Does the assumption in the objective function that "refusals are always judged harmless by the red-team discriminator" hold in the experiments? If the discriminator misclassifies some refusal sentences as harmful, the objective function would introduce systematic bias. Can the misclassification rate on refusal sentences be reported?
  3. In Table 2, GCG-Ours and AutoDAN-Ours have identical numbers in multiple columns (both 44.33 on Llama 3.1 8B, and 100.00 on several models). Is this a coincidence or do they share certain components? If it is a coincidence, can an explanation be added?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.