acceptodds
Under review as a conference paper at ICLR 2027

All you need to break LLMs are Black-Box, Adapting, Efficient, Transferable, Harmful, Applicable … Attacks

19 views

Abstract

Accurately evaluating adversarial robustness is a longstanding challenge. A flawed attack design can inflate robustness estimates, making deployment risk assessment and defense comparison unreliable. Historically, standardized attacks such as AutoAttack have largely resolved this for image classifiers, providing a reliable evaluation baseline for systematic comparison across defenses. However, no equivalent exists for LLM jailbreak evaluation yet, where designing such an attack is considerably more difficult. A reliable attack must, among other things, be black-box compatible, applicable to arbitrary defense pipelines, and efficient, which no existing method jointly satisfies. We introduce Indirect Harmfulness Optimization (IHO), a masked diffusion language model attacker trained via iterative preference optimization against a harmfulness judge, requiring only black-box access to the target. The same method can be used without modification as a strong adapting attack on individual behaviors, or as an efficient amortized policy that transfers to held-out behaviors and unseen target models without fine-tuning. Even against layered defenses, such as GLM 5.3-flash served through a black-box API deploying an auxiliary detector, IHO improves attack success considerably over state-of-the-art approaches, without any defense-specific adaptation. Our results position IHO as a practical step toward the kind of standardized jailbreak evaluation that has improved reliability in the past.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, adversarial robustness, jailbreaking LLMs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fP9pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Indirect Harmfulness Optimization (IHO): a jailbreak method which builds on Inpainting (Lüdke et al. 2025) by adding DPO on feedback from an LLM judge to optimize the diffusion LLM. They also introduce a 6 dimensional framework for holistic evaluation of jailbreaks. Under this framework, they find that IHO has strong transfer across behaviors and is efficient at using few samples at inference time.

Critical strengths

The premise of doing RL to improve the attacker model makes sense, and it seems that this has indeed improved the attack success rate compared to Inpainting. The results on GLM are particularly impressive.

It is good to see a human audit of the attack success labels, particularly since the attacker model was optimized against an LLM judge, and this found minimal reduction in ASR on the undefended models at least.

Critical weaknesses

The attack is not incredibly novel. We have RL trained attackers and diffusion LLM attackers in the literature, this work just combines the two. The six axis framework is particularly well written, but again these are standard considerations in the literature. The level of contribution here feels more like a workshop paper. I would have liked to see a more thorough set of ablations (e.g. are there different ways to apply the attack to a classifier-defended model) and baselines (e.g. AdvGRPO or DC-GRPO) for a conference paper.

I think it is a shame that the authors could not apply their attack to GPT or Claude. I think it makes it harder to support the strong claims around this attack requiring only black box access, transferring across models and being easy to apply.

Questions for authors
  • Where should I be looking for evidence of the claim that attacks transfer well across models with similar defenses? Is it just Llama vs. Qwen32? I would like to see a greater range of models than this. The more impressive transfer is from Qwen to GLM later, but this still leaves a poorly evidenced claim in section 5.2.
  • Figure 5 seems to be missing training step labels? Is (b) cycle 0 vs (c) cycle 6 for instance?
  • It is suspicious that IHO performs better on held-out behaviors. You say this is because they are “slightly easier on average”. If so, it would be good to correct this bias with better sampling.
  • What is going on with the spikes in Figure 5a?
  • I’m not sure what the overall point of section 5.4 is. It seems very obvious that judge scores need to go up during training. I’m not sure why it really matters that output perplexity is going down, or why it is presented in this way rather than say a line plot with training step on the x axis.
  • The most interesting section is 5.5 but only warrants 2 short paragraphs. I would like more analysis here. Why does training on Qwen work better, why does IHO outperform JR1 by so much here when they looked so similar in Table 1, etc.
7 comments

We thank the reviewer for the time invested in reviewing our work and providing feedback.

W1: Novelty

The attack is not incredibly novel. We have RL trained attackers and diffusion LLM attackers in the literature, this work just combines the two. The six axis framework is particularly well written, but again these are standard considerations in the literature.

We agree that the discussed desiderata are not completely novel and acknowledge that our method combines existing ideas. However, a) achieving all six simultaneously is essential for a universally usable method and b) despite being standard considerations, no existing method has managed to satisfy all six simultaneously. We discuss the tradeoffs of many existing options in more detail within the paper. Thus, to the best of our knowledge, the combination of diffusion LLM attacker and offline DPO training makes IHO the first openly available method to achieve them at the same time, which we consider our main contribution. AutoAttack is a good example of a simple (low novelty) method that had positive impact on the field by fulfilling all important attack criteria while providing an easy to use open-source implementation [6].

W2: Ablations and Baselines

I would have liked to see a more thorough set of ablations (e.g. are there different ways to apply the attack to a classifier-defended model) and baselines (e.g. AdvGRPO or DC-GRPO) for a conference paper.

Regarding the suggested baselines, we cannot compare fairly with them, as we could locate neither code nor checkpoints for either attacker. Moreover, DC-GRPO is primarily concerned with credit assignment in a multi-turn setup. We agree that AdvGRPO would be a desirable baseline, but we cannot guarantee within the limited time of this rebuttal that a reimplementation from the paper would not introduce critical mistakes. We therefore keep JR1 as our primary reproducible RL baseline and added Jailbreak-Zero (JB0) as another more recent PO baseline. Please see our response to Reviewer 9JsQ (W1).

Regarding ablations for classifier-defended models: if we understand the question correctly, App. I.3 already contains one such comparison, namely training against the full pipeline versus training against the undefended model and transferring the policy (held-out ASR(512, 0.8), Fig. 20): on CB+D, 0.57 vs. 0.55; on Qwen-2.5-7B+D, 0.65 vs. 0.60. We are happy to add further variants if the reviewer had a different setup in mind.

Action: We added the Jailbreak-Zero results to the paper.

W3: Attacking Models from OpenAI

I think it is a shame that the authors could not apply their attack to GPT or Claude. I think it makes it harder to support the strong claims around this attack requiring only black box access, transferring across models and being easy to apply.

We performed an additional attack evaluation with OpenAI Luna on GuidedBench [1] (see also general comment). Moreover, we are now able to also demonstrate policy transferability for Gemma and Muse finetuned models.

We trained IHO attackers on Gemma-4-31B and on Muse-Glimmer-30B and evaluated each against Gemma-4-31B, Muse-Glimmer-30B and Muse-Spark-Open-816B-A42B, using the same protocol as in the main experiments. Here we used GuidedBench with DeepSeek 4.1 Flash as a judge, which evaluates responses against detailed, behavior-specific rubrics and has an even lower false-positive rate than the judge used in our main experiments [1]. We report the mean harmfulness score, the fraction of behaviors with at least one attack scoring ≥ 0.5, and the fraction fully broken (an attack scoring 1).

AttackerTargetMean scoreBehaviors with attack ≥ 0.5Behaviors fully broken
Gemma0.33792.2%57.2%
trained on GemmaMuse Glimmer0.20177.8%43.9%
Muse Spark0.11156.1%21.1%
Gemma0.28483.3%38.9%
trained on MuseMuse Glimmer0.27980.0%44.4%
Muse Spark0.16461.1%25.0%

These results support our claim that policies learned by IHO are not tied to the model they were optimized against and apply across model families. Moreover, it is evidence of IHO working on the separate dataset GuidedBench that uses a stricter judge policy.

Additionally, we have extended API based results further by attacking Gemini 3.8 Flash and Qwen 3.8 Flash:

TargetMethodJudge ASRPrecision-corrected ASR*
Gemini 3.8 FlashIHO95.0%90.8%
Gemini 3.8 FlashJR155.0%45.6%
Gemini 3.8 FlashInpainting42.5%30.6%
Qwen 3.8 FlashIHO82.5%71.3%
Qwen 3.8 FlashInpainting22.5%8.7%
Qwen 3.8 FlashJR17.5%7.5%

*Rough estimate for the rebuttal only, computed with the precision correction from the submission. See the global comment and our response to Reviewer zFr5 (W1) for its limitations. We will replace it with human-verified ASR.

IHO continues to demonstrate substantial improvements over baseline methods on these newer models.

Action: We added the new transfer experiments, the results on GuidedBench and the additional API runs to the paper.

Q1: Transfer Capabilities of IHO

Where should I be looking for evidence of the claim that attacks transfer well across models with similar defenses? Is it just Llama vs. Qwen32? I would like to see a greater range of models than this. The more impressive transfer is from Qwen to GLM later, but this still leaves a poorly evidenced claim in section 5.2.

Section 5.2 refers to Figure 3 and the full transfer matrices in Appendix I.3 (Figs. 17–22), which cover all 8×8 source–target pairs, not only LLaMA and Qwen-32B. For models with similar defenses, e.g. on held-out behaviors (Fig. 20, ASR(512, 0.8)), the CB-trained policy reaches 0.55 on CB+D, 0.57 on LAT and 0.50 on CAT, while the best baseline reaches 0.20, 0.38 and 0.45. In addition, W3 above reports new cross-family transfer between Gemma-4-31B and the Muse models.

Q2: Figure 5 labels

Figure 5 seems to be missing training step labels? Is (b) cycle 0 vs (c) cycle 6 for instance?

Yes, please excuse the missing annotation. An extended version of (b) and (c) can also be seen in Figure 15. We will extend this plot to remove ambiguity.

Q3: Training split

It is suspicious that IHO performs better on held-out behaviors. You say this is because they are “slightly easier on average”. If so, it would be good to correct this bias with better sampling.

We agree that IHO performing better on holdout data is suboptimal. However, we initially sampled randomly after stratifying by category and only discovered the difficulty imbalance during evaluation. We therefore consider reshuffling the data and retraining after observing performance to be bad practice. To resolve suspicion, we compare all methods on their train and heldout split performance in Table 1. The pattern is widespread: 35 method × target pairs (56.5%) perform better on holdout data, 10 (16.1%) perform equivalently, and only 17 (27.4%) perform worse. IHO performs at least as well on holdout for 6 of 8 targets (75%) and is thus close to the overall rate of pairs performing at least as well (72.6%).

Action: We will note this in the paper to avoid raising suspicion on data leakage.

Q4: Means in Figure 5a

What is going on with the spikes in Figure 5a?

The spikes are the means. We apologize for the missing legend.

Action: We will mark them more clearly with dotted lines and add a small legend to further clarify.

Q5: Section 5.4

I’m not sure what the overall point of section 5.4 is. It seems very obvious that judge scores need to go up during training. I’m not sure why it really matters that output perplexity is going down, or why it is presented in this way rather than say a line plot with training step on the x axis.

As shown by Beyer et al. [2], increasing harmfulness throughout optimization is not guaranteed and most attacks do not increase harmfulness during optimization. Even GCG, which initially improves harmfulness (Figure 6 in the cited work), eventually saturates near the non-refusal threshold, whereas IHO continues shifting the distribution toward more harmful responses. This is important for estimating worst-case model robustness, as discussed in Section 2 (H.1). Lastly, there is clear connection between being low perplexity (in input and output) and being harmful as shown by the high harm regions. A line plot of perplexity and/or harm (y-axis) over time (x-axis) would have failed to show how closely linked they are. Moreover, several existing methods explicitly optimize for low-perplexity attacks [3, 4], partly to evade perplexity-based defenses. IHO achieves decreasing perplexity without explicitly optimizing for it, suggesting that it would 'accidentally' evade such defenses as a byproduct of the DPO optimization.

Action: We will highlight the importance of optimization and perplexity in section 5.4.

Q6: Transfer with weaker model beats direct optimization

The most interesting section is 5.5 but only warrants 2 short paragraphs. I would like more analysis here. Why does training on Qwen work better

The stronger performance of the Qwen-trained policy is consistent with findings in related work [5], where training against weaker models has similarly been shown to improve attack effectiveness. The exact mechanism is an open research problem and slightly out of scope for this paper. The most likely explanation is that denser rewards simply enable more effective attacker training.

Reviewer_9JsQOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Indirect Harmfulness Optimization, a method for training adversarial prompt generators through iterative preference optimization using black-box feedback from target models.

Specifically, the method first employs a parameterized attacker model to generate candidate adversarial prompts. It then constructs preference pairs based on the target language model's responses and the harmfulness scores assigned by an automated judge. These preference pairs are used to update the attacker model through Direct Preference Optimization (DPO). By repeating this process over multiple iterations, the attacker progressively learns to generate prompts with higher attack success rates.

The authors conduct extensive experiments on various open-source language models, adversarially trained defenses, layered safety defense systems, and a commercial black-box API. The results demonstrate strong attack performance across multiple settings, along with promising cross-behavior and cross-model transferability. Additionally, the paper introduces the EVUS metric to address the limitations of conventional Attack Success Rate (ASR) in evaluating attack effectiveness.

Despite the extensive experimental evaluation and promising empirical results, I have significant concerns regarding the methodological novelty, the effectiveness of preference optimization under sparse-reward conditions, the potential for reward hacking of the automated judge, and the fairness of the experimental comparisons.

Critical strengths

S1. Important Research Problem with Practical Significance

This paper addresses the important challenge of effectively evaluating the safety robustness of large language models when only black-box access to the target model is available.

Compared with white-box attacks that rely on access to model gradients or internal parameters, black-box attacks better reflect real-world model deployment scenarios. Therefore, developing automated attack methods that can adapt to different safety defenses, exhibit strong transferability, and maintain reasonable computational costs is of significant practical importance.

S2. Relatively Simple and Generalizable Framework

IHO integrates black-box model feedback, preference data construction, and iterative policy optimization into a unified framework.

Since the training procedure does not require backpropagation through the target model, the framework can be applied to non-differentiable defense components and complex multi-stage safety detection systems.

Furthermore, the framework does not rely heavily on manually crafted attack templates, which enhances its practical applicability.

S3. Comprehensive Experimental Evaluation and Competitive Results

The authors conduct experiments across multiple models and defense configurations, including Circuit Breaker, LAT, CAT, and systems equipped with additional content detectors.

In particular, IHO achieves substantial improvements over several existing methods when attacking models with stronger safety defenses.

The authors also evaluate transferability to unseen target behaviors and target models, providing empirical evidence supporting the generalization capabilities of the proposed approach.

Critical weaknesses

Limited Methodological Novelty and Insufficient Discussion of Closely Related Work

This is my primary concern with the paper.

At its core, IHO employs a parameterized attacker model to generate candidate adversarial prompts, evaluates them based on the target model's responses, constructs preference pairs from high- and low-scoring candidates, and subsequently updates the attacker model using Direct Preference Optimization.

However, the idea of training adversarial prompt generators through preference optimization has already been explored in several prior works.

For example: JailPO (AAAI 2025),RedHit (LLMSEC 2025), ASTPrompter (EMNLP 2025),Jailbreak-Zero (ACL 2026).

The current manuscript does not adequately discuss these closely related works, nor does it include them as primary experimental baselines.

Insufficient Analysis of Cold-Start Challenges and the Effectiveness of Preference Optimization under Sparse Rewards

The paper argues that continuous harmfulness scores enable the attacker to progressively improve its generation capability, even for behaviors that are initially difficult to attack.

However, according to Algorithm 2 in Appendix A, a candidate must exceed a minimum harmfulness threshold to be selected as a positive preference example.

In the main training configuration, this threshold is set to 0.35.

For a given target behavior (b), if all sampled attack candidates satisfy h(b,M(p))<0.35, then no valid positive preference examples can be constructed for that behavior during the current training cycle.

Therefore, IHO does not guarantee that preference optimization can obtain meaningful learning signals when no sufficiently high-scoring candidates are available initially.

Although parameter sharing across behaviors may allow the attacker to learn from easier behaviors and indirectly improve its performance on more difficult ones, this represents a cross-behavior generalization mechanism. It does not directly demonstrate that preference optimization itself resolves the cold-start problem under sparse rewards.

Furthermore, the experimental analysis in Appendix D.1 raises another important concern.

The authors observe that rejected samples in the initial training stage are often empty outputs or semantically incoherent text, whereas chosen samples tend to exhibit greater linguistic coherence.

This suggests that the performance improvements observed during the first DPO training cycle may partly result from the model learning to generate well-formed and coherent prompts, rather than necessarily acquiring stronger adversarial capabilities.

Dependence on an Automated Judge and Insufficient Evidence against Reward Hacking

IHO relies on harmfulness scores provided by StrongREJECT as its primary optimization signal.

However, harmfulness estimated by an automated judge is not necessarily equivalent to the true harmfulness of a model response.

Let the true harmfulness be denoted by

$ H_{\mathrm{true}}(b,y), $

and the score assigned by the judge be denoted by

$ h(b,y). $

The objective actually optimized by IHO is

$ \max_\theta \mathbb{E}[h(b,y)], $

whereas the desired objective is

$ \max_\theta \mathbb{E}[H_{\mathrm{true}}(b,y)]. $

If the judge exhibits systematic biases, the attacker may progressively learn prompt patterns that receive higher judge scores rather than strategies that genuinely elicit more harmful responses.

This concern is particularly relevant because IHO employs an iterative bootstrapping procedure.

In each training cycle, high-scoring candidates are selected as positive preference examples to update the attacker model. The updated model then generates new candidates, which are again evaluated and filtered using the same judge.

If the judge has exploitable systematic biases, this iterative training procedure may repeatedly amplify those biases.

The Relationship between the Proposed Method and the Attacker Architecture Requires Further Clarification

The core preference optimization framework appears to be largely independent of the specific architecture of the generative attacker model.

Appendix F also demonstrates that the same optimization framework can be applied to autoregressive attacker models.

Therefore, I encourage the authors to more clearly distinguish the contributions of the general preference optimization mechanism from those arising specifically from the choice of attacker architecture.

[1]JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs.

[2]RedHit: Adaptive Red-Teaming of Large Language Models via Search, Reasoning, and Preference Optimization.

[3]ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts.

[4]Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models.

Questions for authors

Q1. Novelty

What are the fundamental methodological differences between IHO and existing preference-optimized adversarial prompt generators, such as JailPO, RedHit, and ASTPrompter?

In particular, compared with the iterative DPO framework in RedHit and the online preference optimization approach in ASTPrompter, what new technical contributions does the core optimization mechanism of IHO offer?

Q2. Cold-Start Problem

During the first optimization cycle, what proportion of target behaviors fail to produce any positive preference examples that meet the minimum harmfulness threshold?

Can these behaviors subsequently obtain meaningful learning signals through cross-behavior transfer in later training cycles?

Q3. Learning Mechanism

Can the authors demonstrate that iterative DPO primarily learns generalizable adversarial attack strategies, rather than merely improving the coherence of generated prompts or increasing the likelihood of reproducing high-scoring training samples?

Q4. Reward Hacking

Does the human-verified attack success rate consistently improve as the number of training cycles increases?

Does the false-positive rate of the automated judge change throughout the optimization process?

Q5. Fairness of Experimental Comparisons

Can the authors compare IHO against previous preference-trained adversarial prompt generators under matched end-to-end training budgets and target-model query budgets, and isolate the performance gains specifically attributable to iterative preference data collection and updating?

6 comments

We thank the reviewer for the time invested in reviewing our work and providing feedback.

W1: Methodological Novelty and Related Work

[...] the idea of training adversarial prompt generators through preference optimization has already been explored in several prior works. For example: JailPO (AAAI 2025),RedHit (LLMSEC 2025), ASTPrompter (EMNLP 2025),Jailbreak-Zero (ACL 2026).

The current manuscript does not adequately discuss these closely related works, nor does it include them as primary experimental baselines.

We thank the reviewer for pointing out these related works. We have incorporated a thorough discussion of all four papers into our revised related work section.

While these works share the high-level approach of applying preference optimization to attack generation, two of them address fundamentally different tasks and threat models:

  • ASTPrompter targets prefilling attacks, training an attacker to generate prefixes that induce the target model to produce arbitrary harmful continuations. This differs from standard jailbreaking, which requires eliciting a specific harmful target behavior. Furthermore, ASTPrompter assumes access to assistant prefilling (which is disabled in most commercial black-box APIs) and requires target model log-probabilities for the main method, precluding fully black-box settings.
  • RedHit targets fixed string-matching attacks, optimizing the model to elicit a specific fixed string (e.g., "I hate humans"). This changes the attacker prompt setup, the optimization objective, and the verification pipeline, meaning it cannot be directly applied to open-ended, goal-directed jailbreaking.

The remaining two methods JailPO and Jailbreak-Zero share a similar pipeline. Since JailPO has no publicly available code and Jailbreak-Zero's implementation is missing core parts (raising NotImplementedError), replicating their results is difficult. However, we did re-implement a variant of Jailbreak-Zero (JB0) matching IHO's threat model. We held the number of target queries of the whole pipeline constant across methods and used the same judge. Since attack strength is known to scale with attacker capability [1], we matched attacker model capabilities. Since the instruct model refused to comply with the attacker prompt used in JB0, we chose an abliterated version failspy/Llama-3-8B-Instruct-abliterated.

AttackerQwen2.5-32BLlama-3-8BLlama-3-8B-RR (CB)GLM-5.3-Flash
JR1+0.780.680.170.225
Inpainting0.530.500.250.275
IHO0.900.800.720.925
JB0, abliterated Llama-3-8B attacker0.7230.7880.4320.300

ASR(512, 0.8) on the 40 held-out behaviors (Table 1, right). For JB0, we report the expected ASR at 512 attempts, estimated from 1024 samples. GLM-5.3-Flash: ASR(128, 0.8) as in Fig. 6a, with JR1 and the Qwen-trained IHO policy.

IHO improves upon JB0 on Qwen-2.5-32B, CB and GLM-5.3-Flash and is on par on LLaMA-3-8B. Furthermore, we highlight a key conceptual contribution that sets IHO apart from these works: identifying and overcoming saturation during optimization. While JailPO and Jailbreak-Zero generate a single dataset and train on it once, we show that performance quickly saturates within a single DPO cycle. IHO circumvents this by iteratively generating fresh data from the updated policy. We show that this approach is effective in practice even for very robust models and sparse rewards (e.g., see the Luna experiment in the global comment). This differentiates IHO from prior single-cycle approaches.

Action: We thank the reviewer for helping us to better contextualize our work among previous approaches. We added all works to the related work section of the paper and the JB0 results.

W2: Sparse Rewards

Insufficient Analysis of Cold-Start Challenges and the Effectiveness of Preference Optimization under Sparse Rewards The paper argues that continuous harmfulness scores enable the attacker to progressively improve its generation capability, even for behaviors that are initially difficult to attack. However, according to Algorithm 2 in Appendix A, a candidate must exceed a minimum harmfulness threshold to be selected as a positive preference example. In the main training configuration, this threshold is set to 0.35. For a given target behavior (b), if all sampled attack candidates satisfy h(b,M(p))<0.35, then no valid positive preference examples can be constructed for that behavior during the current training cycle.

While our method, like any other method, does require some signal, with a continuous judge and a moderate threshold of τ = 0.35, this has not been a problem in our experiments: every behaviour had preference pairs by cycle 2 at the latest. The table below shows the proportion of behaviours for many setups which did not have a pair in the DPO set.

SetupNo pair, cycle 0No pair, cycle 1No pair, cycle 2
CAT25.0%6.7%0%
CB+D11.7%1.7%0%
CB6.7%0%0%
LAT3.3%0%0%
Qwen-2.5-7B+D1.7%0%–
Qwen-2.5-32B0%0%0%
Qwen-2.5-7B0%0%0%
Gemma-3-1B0%0%0%
LLaMA-3-8B0%0%0%
GLM-5.3-Flash0%0%0%

Therefore, IHO does not guarantee that preference optimization can obtain meaningful learning signals when no sufficiently high-scoring candidates are available initially.

We have strong reasons to expect that at least some signal will remain available even against stronger future defenses. This expectation is supported by three considerations: a) continued improvements in language diffusion models and, consequently, adversarial prompt generation; b) the fundamental trade-off between safety and utility, which makes it difficult for defenses to eliminate all adversarial signal without compromising usefulness; and c) the recent success of curriculum learning approaches, which suggest that useful signal can be obtained by progressively targeting more challenging behaviors or stronger models. Taken together, these factors suggest that the availability of sufficient signal is unlikely to become a fundamental limitation of our method.

Although parameter sharing across behaviors may allow the attacker to learn from easier behaviors and indirectly improve its performance on more difficult ones, this represents a cross-behavior generalization mechanism. It does not directly demonstrate that preference optimization itself resolves the cold-start problem under sparse rewards.

We would like to reference the "Effectiveness of IHO under sparse signal" section in the global comment because it discusses Luna as a powerful example of highly sparse signal. The section "Cross behaviour bootstrapping" additionally shows that learning on easier behaviours carries over to more difficult behaviours.

Furthermore, the experimental analysis in Appendix D.1 raises another important concern. The authors observe that rejected samples in the initial training stage are often empty outputs or semantically incoherent text, whereas chosen samples tend to exhibit greater linguistic coherence. This suggests that the performance improvements observed during the first DPO training cycle may partly result from the model learning to generate well-formed and coherent prompts, rather than necessarily acquiring stronger adversarial capabilities.

While the initial performance gains may be partly attributed to learning the chat format, IHO keeps improving after the first cycle, with net EVUS gains from cycle 1 to cycle 6 in all four settings (see the per-cycle table in our response to Reviewer zFr5, W2.4.2).

W3: Reward Hacking

IHO relies on harmfulness scores provided by StrongREJECT as its primary optimization signal. However, harmfulness estimated by an automated judge is not necessarily equivalent to the true harmfulness of a model response. [...] If the judge exhibits systematic biases, the attacker may progressively learn prompt patterns that receive higher judge scores rather than strategies that genuinely elicit more harmful responses. This concern is particularly relevant because IHO employs an iterative bootstrapping procedure. In each training cycle, high-scoring candidates are selected as positive preference examples to update the attacker model. The updated model then generates new candidates, which are again evaluated and filtered using the same judge. If the judge has exploitable systematic biases, this iterative training procedure may repeatedly amplify those biases.

We were quite deliberate in our judge selection to reduce the risk of judge hacking and chose StrongREJECT because it was the continuous judge that best met our requirements (App. G.1). Naturally, it is impossible to completely eliminate the risk of judge hacking when using a judge in the optimization pipeline. Therefore, the share of genuine improvement needs to be separated from judge hacking. While we initially hoped to address this by the precision correction with human labels, we have added an extensive human evaluation of the most important empirical result to the global comment. We consider it strong evidence that IHO's advantage over the baselines is not an artifact of the judge.

W4: Importance of Attacker Architecture

The core preference optimization framework appears to be largely independent of the specific architecture of the generative attacker model. Appendix F also demonstrates that the same optimization framework can be applied to autoregressive attacker models. Therefore, I encourage the authors to more clearly distinguish the contributions of the general preference optimization mechanism from those arising specifically from the choice of attacker architecture.

While we believe that the optimization framework is compatible with both diffusion and autoregressive attackers, we consider the choice of architecture not interchangeable. We deliberately chose diffusion LLMs because they allow direct conditioning on the target behavior without additional handcrafted attack templates and enable efficient generation, with as few as 32 tokens sufficient for strong attacks. We separately motivate the choice of attacker architecture, optimization objective, and optimization algorithm in Section 3.1 ("Principled Design Decisions") of the paper. The central contribution of our work is precisely this systematic combination of design choices, rather than either component in isolation.

[1] Panfilov et al., Capability-Based Scaling Trends for LLM-Based Red-Teaming, 2026

Reviewer_zFr5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that LLM jailbreak evaluation lacks a standardized attack comparable to AutoAttack for image classifiers. It proposes six desiderata for such an attack (adversary knowledge, transferability, adaptability, applicability, efficiency and harmfulness) and argues that no existing attack meets all of them.

The proposed method, IHO, trains a masked diffusion language model (LLaDA-8B-Base with a LoRA adapter) as the attacker using iterative DPO. Prompts whose target responses receive high StrongREJECT scores are preferred over low-scoring ones, and the attacker is resampled after each cycle. Only target outputs are needed, so the same procedure applies to defended models, detector pipelines and a commercial API. On held-out JailbreakBench behaviours, the paper reports 0.72 ASR against Circuit Breakers, versus at most 0.25 for the baselines, and a Qwen-trained policy that transfers to GLM-5.3-Flash.

Critical strengths
  • A novel way to train the attacker. The paper turns a diffusion-based prompt generator into a trainable attack policy, using the judge's harmfulness score as the training signal.
    • Using a masked diffusion LM to infill jailbreak prompts comes from prior work (INPAINTING, Lüdke et al., 2025), where the attacker stays fixed. The new step is to fine-tune it with iterative, judge-scored preference optimization.
    • Bidirectional infilling conditions directly on the target behaviour, so no hand-written attacker prompt is needed. The fixed, short generation window (32 tokens) also keeps the cost per prompt under direct control.
    • DPO trains on responses that have already been scored, so no gradients pass through the target. This makes the method fully black-box and lets it run against non-differentiable pipelines such as external detectors and APIs.
  • Simple, fully black-box design.
    • The attacker needs only the target's decoded outputs. No hand-written templates or seed data are required.
    • The same pipeline is applied without modification to adversarially trained models, models wrapped with an external detector, and an API with an undisclosed classifier.
Critical weaknesses

W1 (significant). The "human-corrected" ASR amplifies judge positives instead of verifying them, and it favours IHO over the baselines.

  • Procedure. One author labels at most 100 judge-positive responses per method–target pair. The labels are pooled into a single precision estimate p̂, which is then plugged into 1/B · Σ_z [1 − (1 − p̂)^{K_z}], where K_z is the number of judge positives for behaviour z (Sec. 5.6, App. H).
  • The correction is driven by K_z, not by verified harm.
    • For a single behaviour, 1 − (1 − p̂)^{K_z} approaches 1 as K_z grows, whatever the value of p̂.
    • With p̂ = 0.05, i.e., 95% of judge positives are false, a behaviour with K_z = 60 of 128 attempts still counts as a 0.95 success. A behaviour with K_z = 1 counts as 0.05.
    • The correction can only remove what the judge already counted. For a method that produces many judge positives per behaviour, it removes almost nothing, however low its precision. A method that mainly fools the judge would keep nearly all of its judge ASR after "human correction".
  • The model behind the formula does not fit an optimized attacker.
    • The formula treats every judge positive of a behaviour as an independent draw with a common precision.
    • An optimized policy produces near-duplicate prompts for the same behaviour, so its false positives are correlated. If the errors are fully correlated, the success probability is p̂, not 1 − (1 − p̂)^{K_z}.
    • Precision also varies across behaviours. Because the expression is concave in p, plugging in a pooled p̂ overestimates the average.
    • App. H acknowledges both effects but only argues that estimating p̂ biases the result downward (Jensen). That argument covers sampling noise in p̂ and does not offset the two upward biases above.

W2. The training signal is sparse and unstable, and its limits are not characterised.

  • Signal exists only where the initial attacker already succeeds.
    • A chosen sample must score ≥ 0.35 and lie in the top 3.125% of its behaviour (Tables 13–14). With 15,360 samples per cycle over 60 behaviours, that is at most 8 chosen prompts per behaviour per cycle.
    • Behaviours with no such sample yield no preference pairs at all (Alg. 2). App. D.1 states that "for many combinations, the initial sample pool does not contain enough high-harm chosen candidates".
  • All reported targets are ones where the starting attacker already has a foothold. INPAINTING alone reaches 0.50–0.68 ASR on the undefended models and 0.25–0.32 on CB (Table 1). How the method behaves when the initial success rate is near zero, which is the case a standardized attack must handle, is not shown.
  • There is no per-behaviour evidence for the bootstrapping claim. Sec. 3.1 says that each cycle "progressively covers behaviors the earlier policy could not break". There is no breakdown showing whether behaviours with zero initial positives ever become breakable, or whether the gains come only from behaviours that were already partly broken.
  • The paper's own results show signs of instability.
    • The policy trained directly against GLM is weaker than the policy transferred from Qwen-32B (85.0% vs 92.5%), which the authors attribute to denser reward on the weaker model.
    • The quality-threshold effects in Table 4 are inconsistent across models and change sign (e.g., CB 0.2 → 0.6: −0.156; LLaMA-3-8B 0.2 → 0.7: −0.009). They also rest on only 1–10 behaviours per cell, and App. D.1 notes that the first cycle mostly teaches basic coherence.
    • Within a cycle, gains saturate and then oscillate (App. D.3, Fig. 13).
  • No run-to-run variance is reported. There are no seeds or standard deviations anywhere in the paper, and the ablations use a single-target, single-cycle proxy (App. D).
Questions for authors

Please explain W1, which is a critical concern to me.

4 comments

We thank the reviewer for the time invested in reviewing our work and providing feedback.

W1: Human-corrected ASR formula is not appropriate

We agree with the critique of the precision correction and, being unable to find a suitable alternative, decided to avoid making more distributional assumptions and instead relied heavily on direct human evaluation. Please see global comment "Concerns about judge hacking".

Action: We will remove the precision-corrected ASR metric from the paper. Instead, we added the human-verified results from the global comment to the paper and will conduct the same procedure for the remaining models using Labelbox.

W2.1: Few prompts meet filtering criterion

  • Signal exists only where the initial attacker already succeeds.
    • A chosen sample must score ≥ 0.35 and lie in the top 3.125% of its behaviour (Tables 13–14). With 15,360 samples per cycle over 60 behaviours, that is at most 8 chosen prompts per behaviour per cycle.
    • Behaviours with no such sample yield no preference pairs at all (Alg. 2). App. D.1 states that "for many combinations, the initial sample pool does not contain enough high-harm chosen candidates".

We believe that there is a misunderstanding here. The quoted line exclusively refers to the fact that for a threshold of e.g. 0.7 we were often unable to find at least 16 successful attacks within 1024 attempts, which was the compute budget for the quality experiment. However, this is very different to having no preference pairs at all for the moderate threshold of 0.35. The subsequent table shows, for the 10 behaviours of this ablation, the number of behaviours which for a given (target model, threshold) pair had no chosen sample. Lastly, as noted in "Cross behaviour bootstrapping" in the case of e.g. CAT we could still learn from other behaviors.

Model≥0.20≥0.30≥0.35≥0.40≥0.50≥0.60≥0.70
CAT0/100/101/101/101/103/103/10
CB0/100/100/100/100/101/105/10
CB+D0/100/100/100/100/102/105/10
LAT0/100/100/100/102/103/105/10
Qwen-2.5-32B0/100/100/100/100/100/102/10
Qwen-2.5-7B0/100/100/100/100/100/101/10
Qwen-2.5-7B+D0/100/100/100/101/102/105/10
Gemma-3-1B0/100/100/100/100/100/102/10
Llama-3-8B0/100/100/100/100/100/101/10

Action: Since the text was ambiguous we now clarify this point in the updated paper.

W2.2 Sparse rewards

All reported targets are ones where the starting attacker already has a foothold. INPAINTING alone reaches 0.50–0.68 ASR on the undefended models and 0.25–0.32 on CB (Table 1). How the method behaves when the initial success rate is near zero, which is the case a standardized attack must handle, is not shown.

We provide new results regarding sparse rewards, where we attack very robust models and the attack has little initial foothold in the general comment addressing these concerns. See "Effectiveness of IHO under sparse signal".

W2.3 Bootstrapping claim

There is no per-behaviour evidence for the bootstrapping claim. Sec. 3.1 says that each cycle "progressively covers behaviors the earlier policy could not break". There is no breakdown showing whether behaviours with zero initial positives ever become breakable, or whether the gains come only from behaviours that were already partly broken.

We thank the reviewer for pointing this out. We provide new results on bootstrapping in the general comment, demonstrating that IHO uses bootstrapping to break robust behaviors. See general comment section on "Cross behaviour bootstrapping"

W2.4.1: Transfer policy is superior over direct training on GLM

The paper's own results show signs of instability. The policy trained directly against GLM is weaker than the policy transferred from Qwen-32B (85.0% vs 92.5%), which the authors attribute to denser reward on the weaker model.

We respectfully disagree that the performance difference between the transferred policy and the directly trained policy indicates algorithmic instability. Algorithmic instability would be indicated by training collapse, oscillating updates, or failure to improve over the initialization. In contrast, training IHO directly on GLM-5.3-Flash considerably improves ASR. As detailed in Section 5.5 and Appendix J.2, the GLM-5.3-Flash API deploys an auxiliary content detector that blocks flagged requests, returning empty responses that receive zero reward. When training from scratch against such a discontinuous defense, fewer high-scoring chosen samples above the quality threshold (τ=0.35\tau = 0.35) are available in early cycles. Similar results regarding sparse rewards also have been found in other parts of literature [2].

W2.4.2: Quality-threshold effects are inconsistent

The quality-threshold effects in Table 4 are inconsistent across models and change sign (e.g., CB 0.2 → 0.6: −0.156; LLaMA-3-8B 0.2 → 0.7: −0.009). They also rest on only 1–10 behaviours per cell, and App. D.1 notes that the first cycle mostly teaches basic coherence.

First, Table 4 presents a targeted ablation of preference quality and is not representative of the general stability of our optimization procedure. In fact, EVUS rises steadily across cycles (substantial net gain in all four settings; the only decreases are ≤0.01 and recover the next cycle), as shown below:

ModelC0C1C2C3C4C5C6
Qwen-2.5-7B0.56680.68400.78260.83170.82330.83540.8445
LLaMA-3-8B0.44640.57500.64020.65080.64350.67580.6929
Circuit Breaker0.24390.38080.45120.48770.55020.56750.5867
Gemma-3-1B0.44380.57080.63820.69690.71320.73640.7857

Second, for this ablation, we are reporting the difference between the EVUS scores after training when thresholds differ. In particular, in all four cases performance increased after the first cycle compared to cycle 0! Considering this, the 'instability' of LLaMA should be considered noise. The quoted CB case is a single behaviour and the result can be explained because EVUS accounts for frequency and harm. The model trained with the higher threshold failed to produce a single high harm attack while the lower threshold setup produced multiple partial/moderate breaks.

Action: We contextualized this result in the updated paper.

W2.4.3: Saturating and oscillating gains in cycles

Within a cycle, gains saturate and then oscillate (App. D.3, Fig. 13).

We did not use gradient clipping which might explain the outlier of Gemma-3 (right panel of Fig. 13). It is worth noting that improvements are relative to the previous 40 epochs sample within the cycle. So in particular, whenever a sample under- or overestimates the true mean, some form of 'oscillation' because of regression toward the mean should be expected.

Action: We clarified these observations in the updated paper.

W2.5 Reporting standard deviations

No run-to-run variance is reported. There are no seeds or standard deviations anywhere in the paper, and the ablations use a single-target, single-cycle proxy (App. D).

We acknowledge that we do not report standard deviations across independent training runs, primarily due to the substantial computational costs of training and evaluating adaptive attacks across multiple target models and defense configurations.

We note that this reporting practice is also common in closely related LLM jailbreak research, including PAIR [3], AmpleGCG [4], Jailbreak-R1 [5], and the recent Climbing the Hill [2], which report their main attack success rates without standard deviations across independent runs.

Action: For the camera-ready version, we will report run-to-run variance over 3 seeds across all 6 cycles for Gemma-3-1B.

[1] Schwinn et al. A Coin Flip for Safety. ICML 2026. [2] Yin et al. Climbing the Hill: Prompt Injection Red-Teaming. arXiv:2609.33628, 2026. [3] Chao et al. Jailbreaking Black Box Large Language Models in Twenty Queries. arXiv:2310.08419, 2023. [4] Liao & Sun. AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes. arXiv:2404.07921, 2024. [5] Guo et al. Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning. arXiv:2506.00782, 2025.

Reviewer_zFr5OpenReview ↗

Raised score from 1 to 2

Thanks the authors for their great efforts during the rebuttal period, and I appreciate the human review for those experiment results. I think it's fine to adopt SOTA LLM as the judge (e.g., Claude Opus 5.5). Otherwise, it would bring too much burden to the authors if every example is reviewed by human (as the author mentions, it would take 2 full days).

We thank the reviewer for recognizing the effort invested in the rebuttal. We took the recommendation of using a SOTA LLM as a judge and judged three models across three methods with Kimi K3.1 Pro. We also explored the reviewer's recommendation of using Claude Opus 5.5 and additionally GPT-6.1 Sol and Gemini 4 Carbon, but more than 9 percent of judging requests were blocked by provider-side content filters for all of those options. Since positive jailbreaks can be rare, a judge that fails to judge harmful content is prohibitive. We chose Kimi K3.1 Pro as the most capable model to reliably judge jailbreaking attempts. It scored 55.1 on the Artificial Analysis Intelligence Index (October 2026), ranking 4th of 215 models, within 2.5 points of Claude Opus 5.5, and top 10 under the current Index v5. Different judge thresholds are displayed for transparency.

We rejudged with Kimi and the StrongREJECT rubric prompt, which is widely used in the literature for judging without fine-tuned judge models. The attack data is the same as discussed in the paper or other parts of the rebuttal.

TargetMethodKimi > 0.5Kimi > 0.8Kimi = 1.0Gemma SR > 0.8 (original)
GLM-5.3-FlashIHO100.097.587.592.5
Inpainting77.557.532.527.5
JR180.070.055.022.5
Qwen 3.8 FlashIHO100.097.587.582.5
Inpainting70.035.027.522.5
JR155.047.537.57.5
Gemini 3.8 FlashIHO100.0100.095.095.0
Inpainting95.080.055.042.5
JR195.095.090.055.0

The ASR values under the Kimi judge are generally higher than the results produced by the fine-tuned Gemma-StrongREJECT model. IHO is clearly the best method under all three thresholds, while the gap is narrower for looser thresholds and wider for stricter ones.

Additionally, we report expected ASR values for different sample budgets per behavior to highlight IHO's sample efficiency:

TargetMethodKimi > 0.5: @1 / @8 / @32Kimi > 0.8: @1 / @8 / @32Kimi = 1.0: @1 / @8 / @32
Gemini 3.8 FlashIHO41.3 / 81.7 / 95.932.9 / 76.4 / 93.923.0 / 64.0 / 84.7
Inpainting5.3 / 27.7 / 60.43.6 / 18.9 / 45.42.4 / 11.9 / 29.6
JR117.1 / 57.6 / 84.514.7 / 51.2 / 79.311.1 / 41.4 / 71.1
GLM-5.3-FlashIHO26.5 / 72.1 / 93.113.1 / 51.0 / 82.07.3 / 29.9 / 60.1
Inpainting3.4 / 17.4 / 44.02.0 / 9.1 / 24.91.4 / 6.0 / 14.3
JR16.6 / 26.4 / 50.85.5 / 22.9 / 44.53.9 / 15.9 / 30.1
Qwen 3.8 FlashIHO20.8 / 63.2 / 90.012.1 / 46.6 / 79.76.9 / 28.0 / 59.4
Inpainting2.9 / 14.7 / 38.81.5 / 7.2 / 18.41.0 / 5.2 / 12.6
JR12.9 / 11.6 / 25.42.6 / 10.6 / 22.32.0 / 8.0 / 16.5

IHO achieves the highest expected ASR for every target, threshold and sample budget, and is particularly efficient. On GLM-5.3-Flash and Qwen 3.8 Flash at Kimi > 0.8, IHO with only 8 samples per behavior already exceeds both baselines with 32 samples (e.g., 51.0 vs. 44.5 for JR1 on GLM-5.3-Flash).

We conclude that despite differences in judges, IHO proves to be a harmful and sample-efficient attacker. We would like to note that judging with Kimi K3.1 Pro is not feasible at the scale of our experiments. For the results displayed here, API costs were about $200. Results like our cross-model transfer results would be prohibitively expensive at this cost.

Action: We will extend Kimi judging to more of our core experiments to support our claims across more capable judges.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Comment to all Reviewers

Concerns about judge hacking

We thank the reviewers for raising their concerns about the evaluation protocols used within this paper, as we ourselves are heavily concerned about this topic and agree that providing strong evidence is crucial.

Since false positives in LLM-based judges are a well-documented issue in the literature [1], we adopted the recently published verification protocol of Schwinn et al. [1] for conducting human labeling. However, we have to agree with the mathematical arguments provided by Reviewer zFr5 and being unable to find a better alternative, we decided to exhaustively human-label the most important results according to the following protocol: We evaluated four attack methods (JR1, Inpainting, IHO-Qwen and IHO-GLM) across the 40 training behaviors of the GLM-5.3-Flash experiments (Fig. 6), corresponding to 160 behavior–method pairs and 20,480 generated responses. In order to make manual evaluation tractable, we prefiltered responses using the automated judge threshold of 0.8, yielding 1,822 candidate responses. This follows the established practice of annotating judge-positive responses only, motivated by previous findings that the false-negative rate of harmfulness judges is comparatively low [1]. Consequently, the evaluation focuses on verifying whether automated positive judgments correspond to genuinely harmful outputs. To further cut labeling cost, we labeled each behavior–method pair only until a harmful response was found, which suffices to decide whether the behavior is broken. If no response was harmful, all judge-positive responses of that pair were labeled. This resulted in 824 labels and two full days of annotation. All labels were assigned by one author. For each candidate, the annotator was shown the original harmful behavior and the target model's response and assigned a binary label (harmful or benign). The adversarial prompt and attack-method identity were hidden to minimize potential annotation bias. Responses from different methods were interleaved in a randomized order within each behavior.

MethodLLM Judge ASRPrecision-corrected ASR (paper)Human-verified ASR [95% Wilson CI]Verification rate (Human / Judge)
JR122.5%17.1%10.0% [4.0, 23.1]44.4%
Inpainting27.5%16.1%15.0% [7.1, 29.1]54.5%
IHO-Qwen92.5%86.0%50.0% [35.2, 64.8]54.1%
IHO-GLM85.0%78.1%42.5% [28.5, 57.8]50.0%

Human-verified ASR is the fraction of all 40 behaviors with at least one human-confirmed harmful response among the judge-positive candidates. Verification rate is the fraction of judge-positive behaviors confirmed in this way, equivalently human-verified ASR divided by LLM Judge ASR.

The verification rate is comparable across methods (44.4–54.5%) and is not systematically lower for IHO (50.0–54.1%). Most importantly, the substantial performance advantage of IHO persists under direct human verification (IHO-Qwen 50.0% [35.2, 64.8] vs. 15.0% [7.1, 29.1] for the best baseline). Also note that we did not recompute the precision-corrected ASR with the judge precision p̂ estimated from the new labeled data, because of the bias introduced by early stopping. Moreover, since we now have full human labels for every behavior, this approach would just introduce errors. Lastly, we would like to emphasize that obtaining such a large labelling took two full days, so it could not be extended to more settings during the rebuttal.

Action: We will remove the precision-corrected ASR metric from the paper. Instead, we added the above human-verified results to the paper and will conduct the same procedure for the remaining models using Labelbox.

[1] Schwinn et al. A Coin Flip for Safety. ICML 2026.

AuthorsOpenReview ↗

General Comment to all Reviewers 2

Effectiveness of IHO under sparse signal

Discussion of Luna experiment

To test how IHO performs when the initial success rate is near zero, we ran a new experiment attacking OpenAI Luna (via OpenAI's Researcher Access Program) evaluated on GuidedBench using DeepSeek 4.1 Flash as a judge [2]. This setting is particularly strict for two reasons. First, Luna is a robust model protected by OpenAI moderation flags. Second, GuidedBench evaluates outputs against detailed, behavior-specific rubrics rather than a general judge prompt. GuidedBench was also chosen as the authors have shown that it provides more reliable automatic judge results compared to JBB or HarmBench. We confirmed this by manually labeling 50 responses, all of which agreed with the judge. We report the ASR on GuidedBench below.

IHO Cycle0123456
ASR1.7%2.8%6.7%7.2%8.3%10.0%13.9%

At cycle 0, the base attacker starts very close to zero with a success rate of only 1.7%. Despite this sparse initial signal and active classifier filtering, IHO consistently improves across cycles. Over 6 cycles, the ASR increases by more than 8x (from 1.7% to 13.9%) with no saturation visible. This shows that IHO does not require a strong initial signal and can effectively optimize against heavily guarded models even when starting from near-zero success.

Unfortunately, even though we were given these credits for the purpose of red-teaming OpenAI models, our account was still flagged. In an exchange with OpenAI we agreed to stop evaluations for now. We can share the grant number after the review process to avoid double blind violations.

Action: We added a discussion about the effectiveness of IHO under sparse reward and the Luna experiment to the paper.

[2] Huang et al., "GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods", ICLR 2026

AuthorsOpenReview ↗

General Comment to all Reviewers 3

Cross behaviour bootstrapping

We agree it is beneficial to discuss bootstrapping and generalization in more detail. We provide two major results that provide evidence for bootstrapping in IHO.

First, IHO consistently outperforms Inpainting on held-out behaviors, i.e., behaviors never seen during optimization. Inpainting is essentially equivalent to IHO before its first DPO cycle, apart from differences in attack length and unmasking steps. Thus, every setting in which IHO achieves higher ASR than Inpainting on held-out behaviors (all eight targets in Table 1, right, of the paper) provides evidence that optimization improves performance beyond the behaviors used for training.

Second, we directly compared the original LLaDA attacker (cycle 0) against the attacker after its first DPO cycle (cycle 1). In particular, we measured [email protected] on the training behaviors, using the 15,360 samples that seed the DPO preference dataset, as well as the relative change in mean harmfulness scores on both training and held-out behaviors. If a behavior does not exceed 0.35 in the first cycle, i.e. 25% of behaviours for CAT, then all improvements for cycle 1 for that behaviour can only come from bootstrapping. The same holds for held-out behaviours, which never enter the preference dataset. While we generally observe lower mean harm improvements for held-out behaviours, there is still a clear improvement in mean harm.

SetupTrain [email protected] C0Train [email protected] C1Train mean harm changeHeld-out mean harm change
CAT75.0%93.3%+267.6%+54.3%
CB+D88.3%98.3%+310.9%+400.0%
CB93.3%100.0%+190.4%+56.9%
LAT96.7%100.0%+274.3%+4.9%
Qwen-2.5-32B100.0%100.0%+125.9%+68.3%
Qwen-2.5-7B+D98.3%100.0%+228.6%+49.4%
Qwen-2.5-7B100.0%100.0%+122.7%+12.5%
Llama-3-8B100.0%100.0%+154.2%+62.1%

On the training behaviours, the number of behaviours without any sample ≥0.35 drops from 29 to 5 after a single cycle (columns C0 and C1), although these behaviours contributed no preference pairs in that cycle. We observe that mean harmfulness increases on held-out behaviors in all eight settings after just one DPO cycle, with relative improvements ranging from 4.9% to 400.0%. Since none of these behaviors contributed to the preference dataset, these improvements cannot be attributed to memorizing successful attacks on training behaviors. Moreover, while [email protected] on training behaviors is already saturated in several settings, mean harmfulness still increases considerably. Together, these observations provide direct evidence that IHO's optimization transfers improvements from easier behaviors (already broken with threshold = 0.35) to more difficult behaviours (initially unbroken).

Further analysis reveals that nearly all initially unbroken behaviours become broken according to the traditional threshold of 0.5. Using our more stringent threshold we still observe a non trivial gain on the most difficult behaviours in the dataset.

SettingInitially unbroken by C0 @0.35Broken by C6 @0.5Broken by C6 @0.8
Total across all models29288

Action: We added a discussion concerning cross-behavior bootstrapping to the paper, including these new results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.