acceptodds
Under review as a conference paper at ICLR 2027

TESTING WHETHER A JAILBREAK DEFENSE REDUCES ATTAINABLE RISK

Abstract

Jailbreak defenses are usually evaluated by giving an attack a fixed number of attempts and measuring how often it succeeds. A lower success rate can mean two different things: the defense made some behaviors unreachable, or it only made them take more attempts. A fixed budget cannot tell these apart, so delay can be reported as prevention. We introduce a certificate that separates the two: it replays queries that already succeeded on the undefended model, and every one that still succeeds proves a behavior was not prevented. We validate the certificate on ANCHOR, a benchmark we construct in which each defense’s true effect is known. A standard evaluation reports a large gain for a reversible input recoding that cannot prevent any behavior, whereas our certificate shows it could have prevented at most 3.6% of them. Stochastic filters leave behaviors reachable but increase the attempts required. The certificate also gives informative bounds for four published defenses. Jailbreak evaluations should therefore report prevention and attacker cost separately.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
jailbreak, evaluation, ai safety

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_mkcMOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1(−1)
Soundness
2(−1)
Read review
Paper summary

This work investigates whether improvements in jailbreak defense reflect real prevention of harmful behavior or merely increased attack cost. The authors propose a reachability certificate that replays previously successful queries against defended models to establish an upper bound on attainable-risk reduction. They also introduce ANCHOR, a controlled benchmark with known defense effects, and evaluate the framework on synthetic controls and four existing defenses.

Critical strengths

The paper addresses an important evaluation problem by distinguishing genuine prevention from increased attack difficulty, which is often overlooked in fixed-budget jailbreak evaluations.

The use of negative and positive controls with known ground truth provides a clear experimental setting for validating the evaluation procedure.

Critical weaknesses

The proposed methodology has limited novelty.

The certificate provides a straightforward one-sided bound by counting verified reachable behaviors, without establishing whether uncertified behaviors are actually prevented. The propose problem seems not be solved in this case, as a failed replay is still not be able to prove the defense method block the harmful request.

Moreover, the main experiment uses REPAR, a reversible transformation known to provide no prevention by construction. Thus, the results offer limited evidence of the certificate's added value for evaluating practical defenses.


Although the main claim of prevention vs. delay is somewhat supported during the experiment and also have some metric related insight, it does not provide an new angle in evaluating LLMs' performance in Jailbreaking Success, it is merely just an application of established statistic model.

While it could stands out with a claim to be a diffferent way in characterizing defense effectiveness, it is unclear whether this new metric could leads to more actionable insight for evaluating or improving LLM safety (which is important since metric will be the guidance for future LLM safety research's evaluation and guidance). Since the underlying objective behind this new metric remains unchanged, i.e., reducing the probability of harmful behaviors, the proposed metric also does not provide new guidance to the future LLM training.

Questions for authors

see weakness

1 comment

Thank you for your review.

We agree that a failed replay leaves the behavior unresolved. A successful replay shows that the defense did not prevent that particular behavior. By counting how many behaviors the same attack produces both before and after the defense is added, we can bound how much of the drop in success at a fixed budget could be due to prevention.

We use REPAR to check the method because its true effect is known. Our re-implementation of a paraphrase defense on ANCHOR shows what we can learn when the effect is unknown (Section 6.4, Table 5). At 32 queries per behavior, paraphrasing reduced best-of-n success by 13.5 percentage points. Given more attempts, the same attack produced 222 of 223 behaviors on both systems. Even if paraphrasing prevented the remaining behavior, prevention could explain at most 0.45 percentage points of the drop. That rules out prevention as an explanation for more than 96% of the measured gain.

The SmoothLLM-style result suggests a different step for evaluators. Most sorting tasks remained unresolved, so we tested whether the model could still sort the words when we removed the instruction to withhold the answer. All unperturbed sorting responses were correct. Only 8.7% of responses to individual perturbed inputs were correct. This gives a concrete reason to check ordinary task accuracy before interpreting fewer successful attacks as a safety benefit. These results concern our implementations on benign tasks. Their implications for harmful requests with a released defense and for changes to model training remain open.

Reviewer_3Uv2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that comparing attack success rates at a fixed query budget cannot tell whether a jailbreak defense prevents a behavior (makes it unreachable) or only delays it (lowers the per-attempt success rate). It proposes a "ceiling certificate": queries that succeeded on the undefended model are replayed on the defended model after an adaptation that uses only the defense's public rules, and every verified success shows that the behavior was not prevented. This gives an upper bound on the prevention a defense can claim, and the same replays give a per-behavior attacker-cost multiplier.

The method is validated on ANCHOR, a benign testbed of three exactly verified tasks with control defenses whose true effect is known, mainly on Qwen2.5-7B-Instruct. A standard sweep credits a reversible input recoding with a 0.540 ceiling gain, while the certificate bounds it at 0.036. The paper also reports bounds for four published defenses.

Critical strengths
  • Separating prevention from delay. The paper makes explicit that a lower success rate at a fixed budget can mean two different things: the defense made some behaviors unreachable, or it only made them take more attempts. It reports the two separately (ceiling gain vs. finite-budget gain, Sec. 3; per-behavior attacker cost, Sec. 4.1). Prior work on attack budgets shows that a few attempts can miss failures (e.g., Hughes et al., 2024), but, as far as I know, it does not separate these two effects when reporting a defense's gain. This distinction is worth making, and earlier defense evaluations may well have overlooked it.
Critical weaknesses

W1. Beyond this distinction, the contribution is limited.

  • The certificate is a replay of known successes. For most defense classes, including all four published defenses, the licensed adaptation is the identity (Table 1, Sec. 6.4). The certificate then reduces to checking whether jailbreaks that already worked on the undefended model still succeed at least once in m draws on the defended model.
  • The theory follows from the definitions. Theorem 1 follows from defining every behavior with nonzero success probability to have ceiling 1 (App. A.3), and Props. 1–3 are short arguments.
  • The ANCHOR results hold largely by construction.
    • REPAR's zero gain follows from Prop. 1, and VERUM blocks a fixed fraction of behaviors by design (Table 2).
    • The stochastic filters block each query with probability p, so their cost multipliers (1.99× and 5.00×, Table 4) are simply 1/(1 − p) for p = 0.5 and 0.8.
    • The testbed therefore shows that the method recovers the answers built into the controls, but not what it reveals about real defenses.
  • The cost-multiplier argument is elementary. That the aggregate ratio differs from the per-behavior multiplier follows from E[1/p] ≠ 1/E[p] (Sec. 4.1).
  • Please clarify what the certificate shows that a standard evaluation of known jailbreaks, reported at several budgets, would not.

W2. REPAR, the main demonstration, is not described.

  • REPAR is the paper's headline example (Abstract, Sec. 6.1, Fig. 1b, Fig. 2), but it is described only as a "public bijective input recoding" (Table 2). The paper does not say what the recoding is or which part of the input it is applied to.
  • It is unclear why an unguided attacker succeeds against REPAR at all, and why its success keeps rising with the budget: from ≈0.14 at N = 1 to ≈0.54 at N = 1024 (Fig. 1b), with the parametric fit rejected (p = 0.0026, Sec. 6.1).
  • The attack is not described either. Sec. 5.2 names "unguided resampling" but does not say what varies across attempts or which decoding settings are used.
  • Without these details, it is hard to tell what the 0.540 credited gain measures.
  • Please describe the recoding and the part of the input it is applied to, what unguided resampling varies and the decoding settings, and give an example of a recoded query with the model's response.

W3. The paper does not show that prior evaluations make this mistake.

  • The motivation is that fixed-budget evaluations report delay as prevention. But the statements that defenses are "usually" or "often" evaluated at a fixed budget (Abstract, Sec. 1), and that N0 = 32 is "a typical fixed-budget setting" (Sec. 5.2), cite no such evaluation.
  • Every case where delay passes for prevention involves the authors' own controls: REPAR (Sec. 6.1) and the stochastic filters under best-of-n (Sec. 6.2). REPAR, the main example, is not modeled on any cited defense (Table 1, Table 2).
  • The overturned robustness claims cited in Sec. 1 (Andriushchenko et al., 2025; Nasr et al., 2025) come from stronger, adaptive attacks, not from separating delay from prevention.
  • For the four published defenses, the paper explicitly makes no comparison with their reported gains: "we neither refute them nor estimate how common the measurement failure is" (Sec. 8; also Sec. 6.4, App. B.3).
  • Table 6 suggests natural candidates: on ANCHOR, SmoothLLM and ICD are still rising between N = 32 and N = 1024 (+0.205 and +0.260).
  • Please show at least one published defense evaluation whose reported gain is mostly delay, for example by applying the certificate to the original paper's behavior suite and attack. Without such a case, the practical relevance of the problem is not established.
Questions for authors
  • Please explain what the certificate adds beyond replaying known jailbreaks (W1).
  • W2 and W3 are clarification that need no new experiments; please address them.
1 comment

Yes. I’d keep this response. In W1, I’d change “how much of the observed drop could persist” to “how much of the observed drop prevention could explain.” A delay can persist over many attempts, so the second wording is more precise.

Thank you for your review. We have uploaded the revised manuscript.

W1. A standard evaluation at several budgets can give the same bound if it records which behaviors the attack reaches on each system. Our contribution is to compare that bound with the drop in success at the original budget. Each behavior the same attack produces on both systems rules out prevention for that behavior. Counting those successes therefore limits how much of the observed drop prevention could explain, without predicting where the success curve will level off. We use controls with known outcomes to check this interpretation. For behaviors the attack still reaches, we also estimate how many more queries it takes to succeed.

W2. REPAR shifts each letter in the request by 13 places in the alphabet (ROT13). Digits, punctuation, and the separate record containing the nonce stay unchanged. Unguided resampling sends the same request on every attempt and samples a new response at temperature 1.0, with a limit of 64 new tokens.

For example, “Multiply 18 by 20. Give only the number.” reaches the model as “Zhygvcyl 18 ol 20. Tvir bayl gur ahzore.” One logged response began “Gur ahzore vf 2.” None of the 20 responses contained the correct answer, 360. When the attacker reversed the shift before submission, the model received the original instruction and answered 360 in all 20 attempts.

The early successes come from nonce retrieval. The target string remains visible, and even a refusal discloses it if the model quotes it. By 32 attempts, the attack had recovered every nonce. Multiplication accounts for the subsequent rise in the curve. Success on those tasks rose from 4.5% at 32 attempts to 50% at 1,024, while sorting stayed at zero. The reported 0.540 is the difference between the fitted long-run success rates for the unadapted attack with and without REPAR. An attacker who reverses the shift can always restore the original input, so REPAR’s true prevention gain is zero (Sections 5 and 6.1).

W3. We have not shown that a published defense’s reported gain on its original benchmark was mostly delay. The revised introduction cites specific uses of fixed budgets, including JailbreakBench’s transfer evaluation. Section 5.2 explains why we chose 32 queries for our comparison.

Our closest result comes from testing our implementation of a paraphrase defense on ANCHOR. At 32 queries per behavior, paraphrasing reduced best-of-n success by 13.5 percentage points. Given more attempts, the same attack produced 222 of 223 behaviors on both systems. Even if the defense prevented the remaining behavior, prevention could explain at most 0.45 percentage points of the drop. That rules out prevention as an explanation for more than 96% of the measured gain on these benign tasks (Section 6.4, Table 5). Whether the same holds for the released defense on its original harmful-request suite remains open.

Reviewer_Zfq3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a jailbreak defense makes behaviors unreachable or simply increases the effort needed to obtain them. It proposes replaying successful undefended queries, with adaptations based on the defense’s public rules, to obtain an upper bound on the reduction in attainable risk.

Critical strengths
  1. The distinction between finite-budget resistance and prevention is useful and clearly explained. Reporting reachability and attack cost separately provides a more informative picture than a single attack success rate.

  2. The certificate is simple and interpretable, and the controls with known outcomes help validate the evaluation. I also appreciate that the paper treats unsuccessful probes as unresolved rather than as evidence of safety.

Critical weaknesses
  1. The paper correctly argues that an attack failing within a limited budget does not prove that a defense prevents it. However, it later suggests that two defenses are close to their true long-run performance because further testing finds few additional successes, which seems to rely on the same assumption the paper warns against.

  2. Most of the convincing results come from harmless tasks and defenses constructed specifically for the study, while the experiment on harmful requests does not produce an informative result. This leaves it unclear how useful the method would be for evaluating jailbreak defenses in practice.

  3. The paper measures additional attacker effort by repeating an attack that worked before the defense was added. An attacker could instead switch to another, easier attack, so these results alone do not establish how much harder the defense actually makes attacking the model.

Questions for authors
  1. What evidence supports the conclusion that the results for paraphrase and self-reminder are close to their true long-run performance? How can we distinguish behaviors that have been prevented from those that simply need much more testing to recover?

  2. Could the authors provide an example involving harmful requests where the method changes the conclusion about a published defense? This would help clarify what the method adds to a conventional attack evaluation.

  3. How would the estimated attacker cost change if the attacker could try different attacks after encountering the defense? It would be helpful to know whether the reported increase in effort remains when the attacker adapts its strategy.

1 comment

Thank you for your review. We have uploaded the revised manuscript.

Q1. We removed the claim that paraphrase and self-reminder were close to their long-run performance. Even if few new behaviors appear as the budget increases, the attack could still recover more with further attempts. Under best-of-n, the attack reached 222 of the 223 behaviors on both the undefended and defended models for each defense. That leaves just one behavior per defense whose status we cannot determine. Under resampling, the attack recovered 187 behaviors for paraphrase and 147 for self-reminder, leaving more cases unresolved (Table 5).

Q2. We do not have a harmful-request example where the method changes the conclusion about a released defense. In our JailbreakBench check, the attack reached eight of 50 behaviors on the undefended model and recovered none of those eight after we added the defense. We therefore cannot tell whether the defense prevented those behaviors or the attack needed more attempts (Appendix B.5).

Q3. For the in-context defense, we compared how many queries each attack needed to succeed with and without the defense. The median ratio was 95× when repeating the original query and 4.1× under best-of-n. Replay recovered 136 behaviors in this cost comparison, while best-of-n recovered 222. Because the two estimates cover different behaviors and each uses its own undefended baseline, the difference between them does not tell us how much an attacker would save by switching. We tested each attack separately. The cost for an attacker who decides when to switch during a run remains open (Table 6).

Reviewer_WsCtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that a lower jailbreak success rate does not always mean a defense has prevented the behavior. It may just take the attacker more attempts. The authors propose replaying queries that worked before the defense was added and checking which behaviors still succeed. They test this using a benchmark where they know which defenses actually prevent behaviors.

Critical strengths
  • The distinction between prevention and added attacker cost is useful.
  • The controlled benchmark makes it possible to check whether the evaluation reaches the right conclusion.
Critical weaknesses
  1. I found the paper quite hard to follow. There are several different quantities and bounds, and I often had to go back to work out what each result was actually saying. The main idea seems fairly simple: if a query still succeeds after adding a defense, that behavior has not been prevented. I think the paper could explain much more directly what is new beyond this observation.

  2. The strongest example is a public, reversible input change. The attack initially does not undo the change, so the defense looks effective; once it does, the gain disappears. This makes the problem easy to see, but it also feels like something a basic defense-aware evaluation would catch. I am not convinced the technical contribution is substantial enough based on this example.

  3. The experiments do not do enough to change my view. Most are on constructed benign tasks. The harmful-behavior test gives an uninformative bound, and for the four published defenses the paper does not show a matched evaluation where the proposed method changes the conclusion. Overall, I find the motivation reasonable, but the evidence and contribution are way below the ICLR bar.

Questions for authors
  1. Can the authors state more plainly what the certificate contributes beyond replaying successful queries with knowledge of the defense?
  2. Would a standard defense-aware attack reach the same conclusion in the reversible-input experiment? If so, what is gained by using the proposed framework?
  3. Can the authors show a case on a real jailbreak benchmark and a published defense where the method gives a useful bound and changes the conclusion of a conventional evaluation?
1 comment

Thank you for your review. We revised the paper to make it easier to follow and to explain what each measure tells us.

Q1. Replay shows which behaviors the attack can still produce after the defense is added. We use that count to interpret the drop in attack success at the original query budget. For example, if the same attack produces 99 of 100 behaviors on both systems, the defense could have prevented at most one. Prevention can therefore explain at most one percentage point of the observed drop.

Q2. Yes. A defense-aware attacker can undo REPAR’s public transformation. We chose REPAR because this gives us a case where the correct answer is known. Extrapolating from the unadapted attack suggested that REPAR would lower long-run attack success by 54 percentage points. Once the attacker reversed the transformation, it recovered all 215 behaviors it had reached without the defense. REPAR lets us check whether the evaluation mistakes the effect of sending a scrambled request for evidence of prevention.

Q3. We do not yet have an informative result on harmful requests with a released defense. Our closest result comes from testing our implementation of a paraphrase defense on benign tasks. At 32 queries per behavior, paraphrasing reduced best-of-n success by 13.5 percentage points. With more attempts, the same attack produced 222 of the 223 behaviors on both systems. Even if paraphrasing prevented the last one, that would explain less than half a percentage point of the original drop (Section 6.4, Table 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.