acceptodds
Under review as a conference paper at ICLR 2027

CoFiBreak: A Coarse-to-Fine Diversity Reward for Effective Multi-Turn Jailbreak

Abstract

Automated red teaming trains an attacker large language model (LLM) to elicit harmful responses from a victim LLM, but reinforcement learning can collapse attackers onto a narrow set of prompts when attack success is the only reward. Such a collapse of attack diversity can bias safety evaluation toward repeatedly testing near-duplicate attack patterns rather than the underlying harmful intents. Existing methods often trade off attack success or inference cost for greater content or strategy diversity. We propose CoFiBreak, a coarse-to-fine diversity reward for Group Relative Policy Optimization (GRPO) that improves attack diversity while preserving strong attack effectiveness and low inference cost. The coarse reward broadens which strategies the attacker uses by encouraging strategies underrepresented within each GRPO group, while the fine reward broadens how those strategies are realized by discouraging reuse of the same personas, framings, and output formats after harmful-intent information is removed from prompt embeddings. Diversity rewards apply only to attacks that already exceed an effectiveness threshold, so diversity cannot be gained through ineffective attacks. Since both rewards are computed only during training, CoFiBreak also retains inference efficiency. Trained with Llama3.1-8B-Instruct as the victim, CoFiBreak discovers 1.6 to 2.0 times as many distinct successful fine-grained strategies as the strongest baseline across frontier victims on HarmBench. It reaches 98.8% ASR@10 (attack success rate with 10 generations per intent) on AdvBench with the highest diversity score among all compared methods, which is 2.1 times that of the strongest-attacking baseline. The resulting attacks transfer to unseen frontier LLMs, reaching 71.0% ASR@10 on AdvBench with the unseen victim Claude-Opus-4.6. The code will be released upon acceptance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Jailbreak attack, diversity, AI safety

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_68aCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoFiBreak, a multi-round automated red team framework based on reinforcement learning. By introducing a coarse-to-fine diversity reward mechanism, it improves both attack strategy coverage and content diversity unrelated to attack intent, thereby alleviating the diversity collapse problem in jailbreak attack generation.

Critical strengths

1. This paper proposes a coarse-to-fine diversity reward mechanism that balances the coverage of attack strategies with the diversity of content unrelated to attack intent, achieving a good balance between attack success rate and diversity.

2. The proposed method demonstrates strong cross-model transferability, especially on Claude-Opus-4.6, achieving significant performance improvements compared to existing baseline methods.

Critical weaknesses

1. The reward function design is complex, and its necessity lacks sufficient justification.

This paper combines multiple heuristically designed diverse reward items, but the necessity of this complex design is not adequately demonstrated. Although the authors conducted relevant ablation experiments, the removal of some components actually improved the attack success rate. For example, after removing LEACE in Table 4, ASR@1 increased from 66.7% to 70.7%, and ASR@5 increased from 93.3% to 100%, despite a decrease in diversity. Furthermore, the ablation experiments were small-scale, and existing experiments have not sufficiently demonstrated that this complex reward design has a superior overall effect compared to simpler diverse reward schemes.

2. Experimental validation is insufficient, and some conclusions lack consistent experimental support.

The paper mainly uses across-erase to measure diversity, but within-erase more directly reflects the attack diversity under the same intent, and CoFiBreak did not outperform TROJail on this metric. In addition, there is overlap between the training and evaluation intents of AdvBench, making it difficult to fully verify the method's generalization ability to unseen harmful intents. Meanwhile, Table 3 and Appendix D.1 show that CoFiBreak's attack success rate on some unseen target models is still lower than that of SEMA, indicating that its increased diversity has not brought a stable advantage in attack effectiveness.

Questions for authors

1. Could the specific definitions and calculation methods for evaluation metrics such as ASR@1 and ASR@10 be explained more clearly in the experimental setup?

2. The paper uses different attack success criteria on AdvBench and HarmBench. Could the authors further explain whether the criteria are consistent between the two, and consider using a unified Judge for evaluation to verify the reliability and consistency of the experimental results?

If the authors can adequately address the above questions, I am willing to reconsider and increase the grade for this paper.

5 comments

We thank the reviewer for the detailed reading and for noting CoFiBreak's "good balance between attack success rate and diversity" and "strong cross-model transferability". We address the reviewer's question below.

W1 - Role of each reward component

We would like to clarify that because diversity and ASR trade off against each other (shown in Figure 5 that raising the diversity weight from 0.7 to 1.5 lowers ASR@5 from 93.3% to 73.3%), our goal is to generate diverse attacks while retaining high attack success rather than to maximize either metric alone. We are also encouraged that Reviewer 5zck found the ablations helpful in "showing the role of each term".

1) Full-scale ablation. We agree with the reviewer's observation. To evaluate the components more fully, we rerun the reward component ablations on the full HarmBench (159 intents, 10 generations each) and report ASR@1/5/10 together with both intent-erased diversity metrics, including the within-erase diversity suggested by the reviewer. Note that we regenerate and re-evaluate the full HarmBench results for this ablation, so the anchor's ASR@1/10 differs slightly from Table 3.

VariantASR@1ASR@5ASR@10Within-erase div.Across-erase div.
CoFiBreak (anchor)71.196.297.50.3580.363
Without coarse term68.696.298.10.3290.336
Without fine term77.499.499.40.2650.277
Without LEACE68.696.298.10.3480.342
Global only60.494.396.90.3390.351
Local only75.593.196.90.3280.333

All variants reach similar attack success under ASR@10 (96.9–99.4%, i.e., within 4 of 159 intents). At such close ASR, the anchor achieves the highest diversity among all variants. Additionally, removing LEACE no longer increases ASR at full scale (68.6% vs. 71.1% ASR@1).

We further show the practical benefit of CoFiBreak through our downstream defender experiment (response to Reviewer RVCn and Appendix D.3), where fine-tuning on CoFiBreak attacks yields the highest average safety rate against multi-turn attacks (85.7%, compared with 75.1% for SEMA and 83.9% for TROJail) without a significant loss in utility.

2) Ablations scale. Due to the context limitation, we put ablations of 8 design choices in Appendix E, namely the entropy coefficient, the attack-success gate threshold, the dispersion weight, the number of strategies, the coarse-term divergence, the fine-term similarity signal, the attacker backbone (Tables 15–16), and taxonomy granularity (Table 17).

3) Comparison with simpler schemes. Table 3 in the paper includes baselines that optimize only content diversity (Jailbreak-R1) or only strategy diversity (AutoDAN-Turbo, X-Teaming). On AdvBench with the Llama3.1-8B victim, they reach 72.7–90.4% ASR@10 and 0.19–0.24 diversity, compared with 98.8% and 0.3633 for CoFiBreak. In addition, the single-term ablations show that every single-term variant above also yields lower diversity than the full reward.

W2 - Experimental validation

We thank the reviewer for bringing this suggestion to help us strengthen the evaluation. We are also glad that Reviewer RVCn found the approach "novel and well supported with the experiment and ablation". We clarify each concern below.

1) Held-out AdvBench intents achieve similar ranking. In the paper, we follow SEMA's evaluation protocol [1] to include the AdvBench training split and all baselines are evaluated similarly. To address the generalization concern from the reviewer, we report the results on the 104 held-out AdvBench intents only. The ranking is mostly consistent with Table 3, with CoFiBreak achieving the highest ASR on Llama3.1-8B and Claude-Opus-4.6 as well as the highest diversity.

Held-out AdvBench (104 intents)Llama3.1-8B ASR@1 / ASR@10Claude-Opus-4.6 ASR@1 / ASR@10GPT-5-mini ASR@1 / ASR@10Diversity
ActorAttack5.8 / 28.81.0 / 2.91.0 / 2.90.3077
Crescendo18.3 / 84.67.7 / 27.91.9 / 17.30.3498
GOAT6.7 / 47.10.0 / 1.91.9 / 3.80.3327
X-Teaming29.7 / 93.11.0 / 13.92.0 / 2.00.3393
TROJail30.8 / 84.61.9 / 8.70.0 / 1.90.3410
SEMA75.0 / 94.21.9 / 18.33.8 / 23.10.2360
CoFiBreak78.8 / 100.028.8 / 61.52.9 / 15.40.3546

2) Balancing effectiveness and diversity. We agree with the reviewer's observations on TROJail and SEMA, and we see them as useful illustrations of why balance matters, since each shows a tradeoff between attack success and diversity. We added Figure 8 (Appendix D.2 of the revised paper) to visualize this tradeoff with a scatter plot of ASR@10 against within-erase diversity for every unseen victim and benchmark. SEMA (orange) mostly lies in the lower right (high ASR, low diversity) and TROJail (green) in the upper left (high diversity, low ASR), while CoFiBreak (blue) lies in the upper right and is the only attacker that ranks in the top two on both axes in all eight settings.

The figure also shows why neither metric suffices alone:

  • Diversity also counts failed attacks. TROJail's slightly higher within-erase diversity (0.3500 vs. 0.3438 on AdvBench) comes with much lower ASR (65.8% vs. 98.8% ASR@10).
  • ASR does not penalize repeated attacks. SEMA is stronger on some victims, but it has much lower within-erase diversity (0.1432 vs. 0.3438 on AdvBench) and 1.6–2.0× fewer successful strategies (Figure 3).

3) Diversity metric choice. We agree that within-intent diversity is informative, as it measures whether repeated attempts on the same intent vary. We use across-erase as the main metric because the template collapse we target occurs across intents (the same strategy reused with only the request substituted in §3.2), which within-intent metrics cannot capture.

Q1 - Definition of ASR@1 and ASR@10.

For each intent, we sample k = 10 generations and judge each one. ASR@1 is the fraction of intents for which the first generation succeeds. ASR@10 is the fraction of intents for which at least one of the 10 generations succeeds. We have added these definitions to the paper’s experimental setup (Section 4).

Q2. Unified judge.

As described in Appendix B.1, we follow prior work’s [1] setting and use a separate judge for each benchmark. We moved this to the experimental setup in the main text. Here, we re-judge the HarmBench responses of all multi-turn methods with the GPT-4o-mini judge used for AdvBench. We find that the GPT judge is generally stricter, but the ordering is mostly consistent with Table 3 in the paper, where CoFiBreak remains the strongest on both unseen frontier victims, and SEMA and CoFiBreak remain the top two on Llama3.1-8B.

HarmBench, GPT judgeLlama3.1-8B ASR@1 / ASR@10Claude-Opus-4.6 ASR@1 / ASR@10GPT-5-mini ASR@1 / ASR@10
ActorAttack8.8 / 20.84.4 / 10.11.3 / 6.3
Crescendo30.2 / 73.612.6 / 32.12.5 / 13.8
GOAT11.9 / 49.70.6 / 2.50.6 / 4.4
X-Teaming24.7 / 75.32.6 / 13.00.6 / 6.5
TROJail7.5 / 40.91.3 / 12.61.9 / 3.8
SEMA47.2 / 89.96.3 / 22.01.9 / 13.8
CoFiBreak51.6 / 84.913.2 / 44.04.4 / 17.0

We hope these responses address your concerns, and we would appreciate it if you would consider raising your score.


References

[1] Feng, Mingqian, et al. "SEMA: Simple yet effective learning for multi-turn jailbreak attacks."

Reviewer_RVCnOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This authors propose COFIBREAK for training an attacker model through GRPO with a set newly proposed reward functions that encourage diverse rollouts while maintaining efficiency during training. Two diversity rewards are added to GRPO at advantage function level. A coarse reward is defined for rewarding less occupying strategies in the group. A fine reward is defined for rewarding cosine distance from group peers and from a global buffer of earlier rollouts. Through experiments, the authors show COFIBREAK discover 1.6-2.0 as many distinct successful fine-grained strategies as the strongest baseline across frontier models on HarmBench.

Critical strengths

This paper aims to solve the issue with diversity collapses in adversarial training of the attacker. Two different ways of collapses are well considered in the paper with solutions. The attacker reuses the same technique, or writes every technique the same way. Both are handled through the defined reward functions without tradeoffs. The proposed approach is novel and well supported with the experiment and ablation.

Critical weaknesses
  1. The diversity favored training is not tested on how the collected data can help protect the model from attacks. More diverse harmful prompts is not equivalent to a more powerful dataset for training a safe model on. The missing experiment could actually convert the diverse statement from a intrinsic metric into a real benefit.
  2. All the main numbers are based on a single run. The training randomness in RL is not covered in any of the experiment. In appendix, the authors mentioned about running GRPO with one seed, which doesn't bring any statistical power to the result.
Questions for authors
  1. Do you have any result on how the collected attacks helps with training a safer model? How does the diversity metric correlate with the refusal rate of the trained model on general harmful benchmarks? And how does the over-refusal metric look like?
  2. What is the agreement rate between the 397 strategy labeler and a set of human experts?
1 comment

We thank the reviewer for recognizing that our reward design handles both forms of collapse without tradeoffs and that the approach is "novel and well supported with the experiment and ablation". We address the reviewer's question below.

W1&Q1 - Downstream benefit for training safer models

Similar to the multi-turn safety training setup of X-Teaming [1], we fine-tune Llama3.1-8B-Instruct with LoRA on a fixed benign set plus 500 successful attack conversations from each attacker, which are collected on the AdvBench training intents with a fixed refusal reply. Following prior multi-turn defense work [1, 2], we report the safety rate (SAFE %) averaged over held-out multi-turn attacks (Crescendo and X-Teaming on HarmBench intents, and the human-written MHJ [3]), utility as the average of MMLU and GSM8K, and over-refusal on XSTest [4]. Per-benchmark results are in Appendix D.3.

Training dataSafety ↑ (avg. SAFE %)Utility ↑ (avg.)Over-refusal ↓ (XSTest)
No fine-tuning61.072.07.6
Benign only52.370.58.0
Benign + SEMA75.170.712.8
Benign + TROJail83.970.711.6
Benign + CoFiBreak85.770.511.6

CoFiBreak attacks yield the safest defender, 33.4 points above benign-only fine-tuning. Utility does not differ significantly from benign-only fine-tuning or from the base model. The least diverse attacker (SEMA) yields the least safe defender, but, due to the limited time, with only three attackers we cannot establish a correlation between diversity and safety. Over-refusal metric is the fraction of XSTest's 250 safe prompts that the model fully or partially refuses.

W2 - RL training randomness across seeds

We retrain CoFiBreak with 3 random seeds and report the mean ± std of ASR@10 and diversity on the training-time victim Llama3.1-8B-Instruct below. The standard deviations are small (at most 2.1 points in ASR@10 and 0.007 in diversity), and CoFiBreak’s mean keeps a similar ranking against the baselines as in the paper, indicating that GRPO training with our reward design is relatively stable.

AdvBench ASR@10AdvBench Div.HarmBench ASR@10HarmBench Div.
CoFiBreak (mean ± std, 3 seeds)98.2 ± 0.90.358 ± 0.00796.4 ± 2.10.370 ± 0.007
CoFiBreak (worst seed)97.30.35194.30.363
ADV-LLM93.80.00495.00.089
TROJail65.80.34080.50.366
SEMA97.10.17498.10.197

Q2- Human check of the 397-strategy dictionary

To measure how well the strategy labeler agrees with human judgment, we sample 45 successful attacks on the 3 unseen models shown in Figure 3 (GPT-5-mini, Claude-Opus-4.6 and Gemini-3.1-Pro), yielding 122 turn-label pairs that cover 43 distinct strategies. For each pair, we check whether the assigned strategy, as defined in the dictionary [5], fits the attacker's turn. The agreement is 83.6%, and most of the disagreements are near misses rather than unrelated labels, such as a label that matches only part of its definition or overstates the technique used in the turn.


References

[1] Rahman, Salman, et al. "X-Teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents."

[2] Lu, Xiaoya, et al. "X-Boundary: Establishing exact safety boundary to shield LLMs from multi-turn jailbreaks without compromising usability."

[3] Li, Nathaniel, et al. "LLM defenses are not robust to multi-turn human jailbreaks yet."

[4] Röttger, Paul, et al. "XSTest: A test suite for identifying exaggerated safety behaviours in large language models."

[5] Dabas, Mahavir, et al. "Adversarial Déjà Vu: Jailbreak dictionary learning for stronger generalization to unseen attacks."

Reviewer_5zckOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a RL objective to create an attacker LLM that yields diverse and effective jailbreak attacks. The key idea is to posttrain an attacker LLM using a GRPO objective, and the paper proposes two modifications to the standard GRPO objective. The first is a coarse diversity, which helps ensure that multiple attack strategies are generated in each group. The second is fine-grained diversity, which ensures that the actual prompts themselves are different within each attack strategy. This objective yields diverse and effective attacks, giving impressive ASR results on two attack benchmarks while retaining low inference time.

Critical strengths
  1. This paper has a clear and intuitive idea. It is clear that existing methods suffer from a lack of diversity in generation, as illustrated in Figure 1b and Table 3. To address this, the idea of introducing diversity directly into the GRPO objective makes sense. I especially like the breakdown into coarse/fine grained diversity metrics, and the natural questions I had while reading were addressed in the writing through the three design choices introduced: a) we should only reward diversity if attack is effective b) we should ideally remove the intent part in the embedding to compare fine-grained diversity.
  2. The experiments were thorough, showing the role of each term in the diversity, and also showing ASR and diversity increases compared to baselines. Furthermore, the attacks transfer to other models.
Critical weaknesses
  1. The main weakness is accurately showing whether the intent embedding is accurately capturing "intent" - is there a way to quantify this? The paper would benefit from discussion of this.
Questions for authors

See weakness above.

3 comments

We thank the reviewer for the positive assessment, in particular of the coarse/fine-grained diversity metrics and the thorough experiments. We address the reviewer's question below.

W1 - Quantifying how well is intent erasure

During training, we use LEAst-squares Concept Erasure (LEACE) [1] to remove intent from the prompt embeddings, so that the fine reward measures strategy rather than intent. LEACE guarantees in closed form that no linear classifier can predict the erased concept better than a constant predictor. We verify this empirically with two experiments and add to Appendix A.3 of the revised paper.

1) Intent is no longer predictable after erasure. We train linear and MLP classifiers to predict the intent of CoFiBreak's adversarial prompts on HarmBench [2] (159 intents), with and without erasure, and evaluate on held-out prompts. After erasure, accuracy drops by 87–90 points for both classifiers, indicating that intent can no longer be reliably predicted from the embeddings.

EmbeddingLinear classifier acc.MLP classifier acc.
Before LEACE95.6%92.0%
After LEACE8.2%2.3%

2) Erasure removes intent but keeps strategy. Following Table 1 in the paper, we compute the average cosine similarity of two kinds of prompt pairs after intent erasure: same intent, different strategy and same strategy, different intent. Before erasure (Table 1), prompts with the same intent are more similar; after erasure, this reverses and prompts with the same strategy are more similar under both taxonomies and embedding models.

TaxonomyEmbedding modelSame intent, diff. strategy (before → after)Same strategy, diff. intent (before → after)
Déjà VuQwen3-Embedding-8B0.649 → 0.4940.347 → 0.583
Déjà Vutext-embedding-3-large0.595 → 0.4530.401 → 0.579
PAPQwen3-Embedding-8B0.607 → 0.4670.392 → 0.632
PAPtext-embedding-3-large0.553 → 0.4070.388 → 0.617

Together, the two experiments show that the erased representation retains strategy variation while discarding intent, which is what the fine reward needs.


References

[1] Belrose, Nora, et al. "LEACE: Perfect linear concept erasure in closed form."

[2] Mazeika, Mantas, et al. "HarmBench: A standardized evaluation framework for automated red teaming and robust refusal."

I thank the authors for their response. I will maintain my positive score.

Thank you for your reply and for your support of our work. We are glad our response addressed your concerns.

Comments on the paper

1 comment

We sincerely thank the reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the clear and intuitive coarse-to-fine design that handles both forms of diversity collapse, the thorough experiments that show the role of each reward term, and the good balance between attack success and diversity with strong transfer to unseen frontier models.

In response to the reviewers' feedback, we quantified intent erasure, evaluated the downstream benefit of our attacks for training safer models, and strengthened the evaluation of the reward design and the effectiveness–diversity tradeoff. The revisions are highlighted in green in the updated manuscript PDF. The major improvements are summarized below.

  1. Quantifying Intent Erasure (Reviewer 5zck, Appendix A.3). We trained classifiers to predict intent from the prompt embeddings. Their accuracy drops from 95.6% and 92.0% to 8.2% and 2.3% after LEACE erasure. We also show that after erasure, prompts sharing a strategy become more similar than prompts sharing an intent, so the erased representation keeps strategy while discarding intent.
  2. Downstream Benefit for Training Safer Models (Reviewer RVCn, Appendix D.3). We fine-tuned a defender on attacks from each attacker and evaluated it on held-out multi-turn attacks, over-refusal, and utility. Training on CoFiBreak attacks yields the highest average safety rate (85.7%) without a significant loss in utility.
  3. Full-scale Reward Ablation (Reviewer 68aC, response). We reran the reward-component ablations on the full HarmBench (159 intents, 10 generations each). All variants reach similar ASR@10 (96.9–99.4%), and the full reward achieves the highest diversity on both intent-erased metrics.
  4. Effectiveness–Diversity Tradeoff on Unseen Victims (Reviewer 68aC, Appendix D.2). We added Figure 8, which visualizes ASR@10 against within-erase diversity on four unseen victims and two benchmarks. CoFiBreak is the only attacker that ranks in the top two on both axes in all eight settings.

We thank the reviewers for recognizing the value of our coarse-to-fine diversity reward. We hope the new experiments and clarifications address the main concerns. We appreciate the time and effort the reviewers have devoted to evaluating our work and helping us improve the manuscript, and we are encouraged by their feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.