acceptodds
Under review as a conference paper at ICLR 2027

Closing the Distribution Gap in Adversarial Training for LLMs

Abstract

Adversarial training has been one of the most promising methods to reliably improve robustness against adversaries and has been recently extended to LLMs. However, despite significant progress, models remain vulnerable to trivial exploits, such as rephrasing the same request or repeated sampling. We argue that this persistent fragility stems from a fundamental limitation in current adversarial training algorithms: they minimize adversarial loss on their training set but inadequately cover the data distribution, resulting in vulnerability to seemingly simple attacks. To bridge this gap, we propose Distributional Adversarial Training, . We leverage Diffusion LLMs to approximate the true joint distribution of prompts and responses, enabling generation of diverse, high-likelihood samples that address generalization failures. By combining optimization over the data distribution provided by the diffusion model with continuous adversarial training, DAT achieves substantially higher adversarial robustness than previous methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial training, LLMs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wz9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Distributional Adversarial Training (DAT), which generates prompts conditioned on harmful responses using a diffusion LLM, filters them based on the target model’s responses, and applies continuous adversarial training. Experiments on three instruction-tuned LLMs show substantial robustness improvements across multiple attacks.

Critical strengths

1.The empirical robustness gains are substantial. DAT reduces worst-case ASR from 0.88 to 0.34 on Llama3 and from 0.98 to 0.18 on Qwen2.5 compared with CAT, and improvements persist after excluding Inpainting (Table 1).

2.The ablations provide useful evidence for the design choices. The diffusion-only and PAIR-DAT controls support the combination of diffusion-generated data and continuous adversarial training, while additional experiments examine prompt filtering and robustness across diffusion attack models (Table 1; Appendices C.2, C.5, and C.7).

Critical weaknesses
  1. The sampling design is not well connected to the stated distributional objective. DAT retains the 12 highest-scoring prompts out of 1,000 candidates per behavior, creating a model-dependent training distribution that is not covered by Theorem 4.1. It remains unclear whether the gains reflect better distributional coverage or more effective attack selection (Section 4.6; Table 11).

  2. The methodological novelty over the closest prior work appears limited. DAT follows the diffusion-based prompt generation procedure of Lüdke et al., while MixAT already combines generated adversarial prompts with continuous adversarial training (Section 2; Appendix C.1). The main methodological change appears to be the prompt generator within this existing training framework.

  3. The broader robustness–utility advantage is not established at matched benign compliance. For example, DAT lowers Qwen2.5’s XSTest score from CAT’s 0.544 to 0.436 (Table 1). The Pareto analysis only covers Inpainting on Llama3, leaving unclear whether the multi-attack robustness advantage persists at comparable compliance across models (Figure 3).

Questions for authors
  1. How does selecting the highest-scoring prompts improve the intended distributional approximation? Can the authors distinguish the benefit of broader coverage from that of selecting more effective attacks?

  2. What is the substantive methodological contribution of DAT beyond combining the existing diffusion-inpainting generator with CAT? Please clarify the advance over MixAT beyond the choice of prompt generator.

  3. Can the authors report WCA–XSTest trade-offs across all three models and compare DAT with CAT at matched XSTest compliance? This would clarify whether the broader robustness advantage persists when benign compliance is controlled.

2 comments

Dear Reviewer wz9u, Thank you for the thoughtful review and for recognizing the substantial robustness gains. We answer the novelty concerns (W2 / Q2) and matched-compliance comparison (W3 / Q3) in the general comment.

W1 / Q1: Coverage vs. selection of effective attacks.

DAT retains the 12 highest-scoring prompts out of 1,000 candidates per behavior, creating a model-dependent training distribution that is not covered by Theorem 4.1. It remains unclear whether the gains reflect better distributional coverage or more effective attack selection (Section 4.6; Table 11).

How does selecting the highest-scoring prompts improve the intended distributional approximation? Can the authors distinguish the benefit of broader coverage from that of selecting more effective attacks?

In Section 6.5 we have ablated the filtering strategy between 12 highest-scoring and randomly sampled candidates with full table in Appendix C.7. The latter has a WCA of 0.60 on Llama3-8B, worse than top-12 but better than all other baselines.

Broader coverage is needed at same training effort. We have another ablation on the broader coverage in Appendix C.2 with an autoregressive attack generator PAIR-DAT. It applies the same top-12 selection to PAIR prompts under the same training recipe and budget. Due to the filtering, the selected prompts for PAIR-DAT and regular DAT also share equal attack effectiveness. PAIR-DAT is robust against PAIR but not GCG and Inpainting, similar to MixAT. Selecting effective attacks alone therefore does not produce the robustness of DAT.

Selection becomes unnecessary with more samples. Furthermore, we followed the suggestion of Reviewer hwjg and added an extra experiment containing a larger set of training data. We observe that random-16 reaches the same level of robustness as top-16 at the same XSTest level:

Llama3-8BGCG ↓GCG+ ↓PAIR ↓BoN ↓Inpainting ↓WCA ↓XSTest ↑
DAT-top160.080.130.090.000.320.360.464
DAT-random160.080.130.100.000.320.330.476

Filtering therefore compensates for a small sample, in which a random sample covers fewer of the rare prompts that elicit harmful responses. Theorem 4.1 describes exactly this unfiltered estimator. At 16 samples per behavior, random sampling follows the theorem and removes the dependence of the training set on the target model. We plan to run the same comparison on Qwen2.5-14B and Gemma3-12B and further investigate the role of filtering.

Action: We add the 16-sample comparison to Appendix C.7 and revise Section 6.5 to state that filtering matters only at small sample sizes.

W3 / Q3: WCA–XSTest trade-off at matched compliance.

Can the authors report WCA–XSTest trade-offs across all three models and compare DAT with CAT at matched XSTest compliance? This would clarify whether the broader robustness advantage persists when benign compliance is controlled.

We trained additional DAT models to match the XSTest compliance of CAT on each model (details in the general comment, Tables R1a–c). On the first 40 JailbreakBench behaviors:

ModelDefenseXSTest ↑PAIR ↓GCG ↓Inpainting ↓WCA ↓
Qwen2.5-14BCAT0.5440.2500.1250.9250.925
Qwen2.5-14BDAT0.5280.1750.1250.2500.425
Llama3-8BCAT0.4640.2500.1000.9000.900
Llama3-8BDAT0.4920.2000.0750.2250.350
Gemma3-12BCAT0.4880.5500.4500.9000.900
Gemma3-12BDAT0.4960.1750.1500.0000.300

WCA is taken over the three attacks. At matched or higher XSTest compliance, DAT lowers WCA from 0.925 to 0.425 on Qwen2.5-14B, from 0.900 to 0.350 on Llama3-8B, and from 0.900 to 0.300 on Gemma3-12B. The robustness advantage therefore persists when benign compliance is controlled.

Action: We add these matched-compliance comparisons for all three models to the revision.

Thank you again for the constructive feedback. We have added the random-16 versus top-16 comparison, matched-compliance results on Qwen2.5-14B and Llama3-8B, and a clarification of the contribution over MixAT. We hope these revisions address your concerns and are happy to provide further clarification if needed.

Dear Reviewer wz9u,

We would like to thank you once again for your thoughtful review and for recognizing the substantial robustness gains of DAT and the value of our ablations.

In response to your review, we provided the following clarifications and additional experiments, most of which are now included in the revised paper:

  • Disentangled distributional coverage from attack selection: we pointed to our random-vs-top-12 filtering ablation and to PAIR-DAT, which uses the same top-12 selection but fails to generalize to other attack families (App. C.2, now on all 100 JailbreakBench behaviors).
  • Added a new experiment showing that with 16 samples per behavior, random sampling without any target-model selection matches top-16 filtering (WCA 0.33 vs. 0.36 on Llama3-8B), in line with the unfiltered estimator of Theorem 4.1 (App. C.7).
  • Clarified the contribution beyond MixAT in the general comment: DAT shapes the outer-loop training distribution rather than strengthening the inner-loop adversary, and a single offline diffusion source outperforms the multi-attack MixAT+GCG at roughly 1/8 of its training compute.
  • Compared DAT with CAT at matched XSTest compliance on all three models, where DAT lowers WCA from 0.925 to 0.425 (Qwen2.5-14B), 0.900 to 0.350 (Llama3-8B), and 0.900 to 0.300 (Gemma3-12B) (App. C.8, Table 13).
  • Added a WCA–XSTest trade-off plot for all three models (Fig. 4).

As we approach the end of the discussion period, we would appreciate hearing whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining questions or provide further clarifications.

Reviewer_5pWQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Distributional Adversarial Training (DAT) to improve the generalization of safety-oriented adversarial training for LLMs. DAT uses a diffusion language model to generate prompts conditioned on affirmative harmful responses, filters candidates using the target model’s responses and StrongREJECT scores, and applies continuous adversarial training to the selected prompts. A KL regularizer on benign data is used to preserve utility. The paper motivates this approach by distinguishing limitations in training-distribution coverage from limitations in inner-loop adversarial optimization. It evaluates three model families against seven attack types and reports substantial reductions in aggregate worst-case attack success rates. Additional experiments investigate generator choice, sample count, filtering, training modifications, and transfer across diffusion attack models.

Critical strengths
  1. Substantial empirical improvements under a diverse attack suite. Table 1 reports WCA reductions from the strongest listed comparator’s 0.88 to 0.34 on Llama3, 0.65 to 0.18 on Qwen2.5, and 1.00 to 0.19 on Gemma3. The advantage also persists against the more expensive MixAT+GCG baseline in Table 8. Evaluating both optimization-based and sampling-based attacks, including Claudini, makes the results more informative than evaluations restricted to a narrow attack family.
  2. Useful evidence that both generated data and continuous optimization matter. The Diff-only results, filtering ablation, and sample-count analysis help explain the method’s behavior. The PAIR-DAT control in Appendix C.2 and the additional diffusion attack models in Appendix C.5 are particularly useful attempts to test whether the benefits extend beyond a single generator or attack family. These experiments provide a meaningful empirical case for further studying diffusion-generated data for safety training.
Critical weaknesses
  1. The incremental methodological contribution and its causal attribution remain insufficiently established. MixAT [1] already combines generated discrete adversarial prompts with continuous perturbations centered on those prompts. Continuous adversarial training itself is established in [2], while diffusion-based conditional generation of diverse, transferable adversarial prompts is introduced in [3]. Thus, DAT’s central contribution appears to be the use of diffusion-generated and filtered data within an existing training paradigm, together with an empirical and distributional justification. The paper acknowledges these connections and includes relevant comparisons; my concern is therefore not missing citations. Rather, the evidence does not yet sufficiently isolate the contribution of the diffusion generator from candidate-generation budget, filtering, and training choices. Appendix C.2 is helpful, but does not clearly establish matched candidate-generation and selection budgets for PAIR-DAT and DAT, and evaluates only 40 behaviors. A stronger controlled comparison is important because generator choice is central to the claimed contribution. The fidelity bound provides motivation, but, as the paper appropriately acknowledges, does not guarantee the behavior of the filtered finite-data implementation.
  2. The practical robustness–utility advantage is not yet established consistently across models and the full attack suite. The headline WCA improvements are substantial, but the selected DAT configurations also reduce XSTest scores relative to the undefended models: 0.660 to 0.424 for Llama3, 0.748 to 0.436 for Qwen2.5, and 0.696 to 0.443 for Gemma3. Qwen2.5 additionally loses 5.2 percentage points on ARC-E. These comparisons do not establish that DAT is worse than other defenses, but show why matched-utility comparisons are necessary. Figure 3 partially addresses this concern, but focuses on Llama3 and Inpainting robustness rather than aggregate WCA across all three models. The distinction matters: when Inpainting is excluded, Llama3’s advantage over the strongest Table 1 comparator is 0.24 versus 0.17, appreciably smaller than the headline WCA difference. Without broader matched-utility results and variability estimates, it remains unclear how consistently DAT improves the overall safety–utility frontier.

[1] Csaba Dékány et al. MixAT: Combining Continuous and Discrete Adversarial Training for LLMs. 2025. [2] Sophie Xhonneux et al. Efficient Adversarial Training in LLMs with Continuous Attacks. NeurIPS 2024. [3] David Lüdke et al. Diffusion LLMs are Natural Adversaries for any LLM. 2025.

Questions for authors
  1. How many independent training runs underlie the main results, and are variability estimates available? Also, does Table 5 exclude candidate generation, target-model inference, and StrongREJECT scoring? Please report or estimate these components separately for the main top-12 procedure.
  2. How many candidate prompts per behavior were generated and scored for PAIR-DAT versus DAT? Were target-model queries, judge calls, filtering rules, and hyperparameter-search budgets matched? Please clarify which differences remain beyond generator choice.
3 comments

Dear Reviewer 5pWQ, Thank you for the careful review and for recognizing our strong empirical results. We answer the matched-compliance comparison (W2) and the contribution beyond MixAT (W1) in the general comment.

W1 / Q2: Matched budgets in PAIR-DAT.

[...] the evidence does not yet sufficiently isolate the contribution of the diffusion generator from candidate-generation budget, filtering, and training choices. Appendix C.2 is helpful, but does not clearly establish matched candidate-generation and selection budgets for PAIR-DAT and DAT, and evaluates only 40 behaviors.

How many candidate prompts per behavior were generated and scored for PAIR-DAT versus DAT? Were target-model queries, judge calls, filtering rules, and hyperparameter-search budgets matched? Please clarify which differences remain beyond generator choice.

Thank you for this comment. We agree that this should be further clarified in the paper. Yes, the budget of each training run was matched, and the generator is the only difference. We report the full budget of both experiments below.

DATPAIR-DAT
GeneratorLLaDA-8BPAIR with Vicuna-13B
Candidates generated and scored per behavior1000identical
Selection ruletop-12 per behavior by StrongREJECTidentical
Retained prompts1200identical
Training recipe6 epochs, 20 attack steps, lr 1e-4identical
Training runs on this recipe611

The hyperparameter search for PAIR-DAT used more runs than DAT, to make sure the gap does not come from insufficient tuning. We further extended the evaluation to all 100 JailbreakBench behaviors: PAIR-DAT is robust to PAIR itself (0.00) but reaches 0.85 Inpainting and 0.70 GCG+ ASR, against 0.24 and 0.06 for DAT (Table 1).

Action: We will add the budget table to Appendix C.2 and update PAIR-DAT result in Appendix C.2.

W2: Robustness–utility advantage across models and without Inpainting.

The practical robustness–utility advantage is not yet established consistently across models and the full attack suite. [...] Qwen2.5 additionally loses 5.2 percentage points on ARC-E. [...] when Inpainting is excluded, Llama3's advantage over the strongest Table 1 comparator is 0.24 versus 0.17, appreciably smaller than the headline WCA difference.

For the matched-utility comparison, we trained additional DAT models and compared them with CAT at matched XSTest compliance in the general comment (Table R1). On Qwen2.5-14B, at similar compliance (0.528 vs. 0.544 for CAT), DAT lowers the WCA from 0.925 to 0.425, with ARC-E at the level of CAT (0.854 vs. 0.857).

Moreover, following Reviewer hwjg, we evaluated an additional attack that no defense saw in training, Jailbreak-R1 (JR1; Guo et al., 2025), an RL-trained attacker LLM (details in our response to Reviewer hwjg, Q3). On the first 40 behaviors of Llama3-8B:

JudgeNo defenseCATDAT
StrongREJECT0.5250.4250.175
HarmBench0.5750.2500.100

CAT, the strongest competing defense without Inpainting in Table 1, is broken by JR1 on 0.425 of the behaviors, more than twice the rate of DAT (0.175; HarmBench 0.250 vs. 0.100). Its low WCA without Inpainting therefore reflects robustness against the specific attacks in our evaluation rather than broad robustness, suggesting that adding further attacks to the evaluation could widen the gap between DAT and the baselines.

Action: We will add these results to the paper.

Q1 (a): Number of independent training runs and variability.

How many independent training runs underlie the main results, and are variability estimates available?

Each model in Table 1 comes from one training run. To reach a target XSTest level, we made at least five training attempts per model, which motivated the Pareto plot in Fig. 3. In the table below, the WCA gaps exceed the sampling uncertainty by a wide margin on all three models. WCA with 95% intervals over the 100 behaviors:

ModelCATDAT
Llama3-8B0.88 [0.80, 0.93]0.34 [0.25, 0.44]
Qwen2.5-14B0.98 [0.93, 0.99]0.18 [0.12, 0.27]
Gemma3-12B1.00 [0.96, 1.00]0.19 [0.13, 0.28]

The intervals of DAT and CAT do not overlap on any model.

To estimate training variability, we evaluated three independently trained DAT models on Llama3-8B with different retain weights and batch sizes, on the first 40 behaviors:

Llama3-8BXSTest ↑GCG ↓PAIR ↓Inpainting ↓
CAT0.4640.1000.2500.900
DAT, retain weight 0.45, batch 8 (paper)0.4240.0500.1750.225
DAT, retain weight 0.85, batch 80.4920.0750.2000.225
DAT, retain weight 0.85, batch 640.4440.0750.1000.375

Across the three runs, Inpainting ASR stays between 0.225 and 0.375, and GCG and PAIR ASR vary by at most 0.10. The advantage of DAT therefore does not depend on one favorable run. The matched-compliance Qwen2.5-14B models in the general comment (Table R1) confirm this on a second model.

Action: We add 95% intervals to all WCA entries and the three-run table to the appendix.

Q1 (b): Cost of generation, target inference, and StrongREJECT scoring.

Also, does Table 5 exclude candidate generation, target-model inference, and StrongREJECT scoring? Please report or estimate these components separately for the main top-12 procedure.

Yes, Table 5 reports defense training only. We now report compute time for preprocessing. Diffusion generation (step 1) is model-independent: one pool of 1000 candidates per behavior takes about 5h and is shared by all three models. The model-specific steps take:

StepLlama3-8BGemma3-12BQwen2.5-14B
(2) Target responses1h43min2h31min38min*
(3) StrongREJECT scoring35min36min4min*
(4) Top-12 selection3sec2sec0.3sec*

*Qwen2.5-14B scored only 100 candidates per behavior from the shared pool due to time constraints.

For comparison, MixAT+GCG needs 12h50 of training alone on Llama3-8B (Table 5), without the cost of its API calls. Based on our newest result, steps (2), (3), and (4) are also not required for DAT. Following the suggestion of Reviewer hwjg, we ran a new experiment that trains DAT on 16 random diffusion samples per behavior without any selection. It reaches the same robustness as top-16 selection (WCA 0.33 vs. 0.36 on Llama3-8B; see our responses to Reviewer hwjg, Q1, or Reviewer wz9u, Q1). These are the first results on Llama3-8B, and we are running the same comparison on Qwen2.5-14B and Gemma3-12B and further investigating the role of filtering. Generating these 16 prompts per behavior takes about 5 minutes at the rate of step (1), so this variant costs almost nothing beyond CAT.

Action: We extend Table 5 with the generation, scoring, and selection costs.

Thank you again for the constructive feedback. We have added the full PAIR-DAT budget, matched-compliance results on Qwen2.5-14B, intervals and a three-run variability table, and the complete cost breakdown. We hope these revisions address your concerns and are happy to provide further clarification if needed.

Dear Reviewer 5pWQ,

We would like to thank you once again for your careful review and for recognizing our results under a diverse attack suite and the evidence that both the generated data and continuous optimization matter.

In response to your review, we provided the following clarifications and additional experiments, most of which are now included in the revised paper:

  • Reported the full PAIR-DAT budget, showing that candidate generation, scoring, selection, retained prompts, and training recipe are identical to DAT, with the generator as the only difference, and extended the PAIR-DAT evaluation to all 100 JailbreakBench behaviors (App. C.2, Table 7).
  • Clarified the contribution beyond MixAT in the general comment: DAT targets the outer-loop training distribution rather than strengthening the inner-loop adversary, and a single offline diffusion source outperforms the multi-attack MixAT+GCG at roughly 1/8 of its training compute.
  • Compared DAT with CAT at matched XSTest compliance on all three models, including matched ARC-E utility on Qwen2.5-14B (App. C.8, Table 13), and added a WCA–XSTest trade-off plot for all three models (Fig. 4).
  • Evaluated an unseen RL-trained attacker (Jailbreak-R1), which breaks CAT on 0.425 of behaviors versus 0.175 for DAT, indicating that CAT's low WCA without Inpainting does not reflect broad robustness (App. C.10).
  • Added 95% intervals for all WCA entries, which do not overlap between DAT and CAT on any model, and a variability study over three independently trained DAT models (App. C.11).
  • Provided a full cost breakdown of generation, target-model inference, and StrongREJECT scoring, and showed that the new random-sampling variant skips the target-model-dependent steps entirely (App. B.4).

As we approach the end of the discussion period, we would appreciate hearing whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining questions or provide further clarifications.

Reviewer_hwjgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that adversarial training for LLMs mainly reduces the inner optimization error on a fixed training set. It leaves a gap to the population robust risk, because the training prompts cover the space of harmful requests poorly. To close this gap, the authors propose Distributional Adversarial Training, DAT. A diffusion LLM is conditioned on harmful target responses and inpaints many candidate prompts. The prompts that are most effective against the undefended target model are kept, and continuous adversarial training with a KL retain loss is then applied to them. Experiments on Llama3, Qwen2.5 and Gemma3 show a large drop in worst-case attack success rate.

Critical strengths
  1. *DAT achieves the best worst-case robustness on all three models. The gains extend to attack families never seen during training and to inpainting attacks built with other diffusion models. Training cost is comparable to CAT and MixAT and far below MixAT with online GCG.
  2. The diffusion-only ablation shows that both the diffusion data and the inner adversarial loop are needed. Replacing the diffusion model with an autoregressive attacker mainly protects against that attacker's own family. The paper also checks train–test prompt overlap and studies how the number of samples and the filtering step affect robustness.
Critical weaknesses
  1. Theorem 4.1 is a direct total-variation inequality, and its assumption is never measured. The authors acknowledge that filtering, finite data and the unclipped loss fall outside the theorem, so the theory serves as motivation only.
  2. Novelty is moderate. The method combines an existing diffusion inpainting attack with CAT, which is close in spirit to MixAT with a different prompt generator.
  3. DAT reduces XSTest compliance compared with the undefended models and with CAT, which points to more over-refusal. The Pareto analysis covers only Llama3 and measures robustness against inpainting, the same attack family used to build the training data. A trade-off based on worst-case robustness without inpainting would be more convincing.
Questions for authors
  1. Can you separate distributional coverage from attack strength? For example, compare generators at matched attack success of the selected prompts, train on a larger pool of unfiltered diffusion samples, and report a direct measure of how well the generated prompts cover held-out natural harmful prompts
  2. Can you score attacks with a judge different from the one used for filtering, and report the robustness–utility trade-off
  3. How does DAT hold up against adaptive attacks designed with knowledge of the defense, and against multi-turn jailbreaks
3 comments

Dear Reviewer hwjg, We thank the reviewer for the detailed review. We point the reviewer to the general comment regarding novelty (W2) and trade-off (W3).

W1: Theorem 4.1 serves as motivation only.

Theorem 4.1 is a direct total-variation inequality, and its assumption is never measured. The authors acknowledge that filtering, finite data and the unclipped loss fall outside the theorem, so the theory serves as motivation only.

We agree that Theorem 4.1 serves as theoretical motivation, as we note when stating its scope in Section 4.4 and Appendix A.3. At the same time, we run several experiments to check whether its main takeaway, that drawing more prompts from the conditional surrogate without selection brings the empirical robust risk closer to the population robust risk, holds in practice.

  1. Sample scaling: Inpainting ASR on Llama3-8B falls from 54% to 22% as the number of samples per behavior grows from 1 to 16 (Fig. 3, right).

  2. Unfiltered training (closer to the theorem's setting): We thank the reviewer for suggesting we test this closer to the theoretical assumptions. Training on a larger pool of unfiltered diffusion samples matches the performance of top-score filtering at 16 samples per behavior (WCA 0.33 vs. 0.36 on Llama3-8B; see Q1b).

  3. Generator ablation: Replacing the conditional diffusion generator with PAIR under identical training removes most of the gain (Appendix C.2).

We thus argue that the theory serves a practical purpose.

Q1: Separate distributional coverage from attack strength.

Can you separate distributional coverage from attack strength? For example, compare generators at matched attack success of the selected prompts, train on a larger pool of unfiltered diffusion samples, and report a direct measure of how well the generated prompts cover held-out natural harmful prompts

A1: We thank the reviewer for suggesting this setup to analyze different possible contributing factors.

a) Matched attack success generators: Because both training sets pass the same top-12 StrongREJECT filter, the PAIR-DAT training data reaches the same level of ASR as the DAT training data. PAIR-DAT therefore directly implements the suggested comparison of generators at matched attack success. This experiment, as reported in Appendix C.2, shows that PAIR-DAT increases robustness to PAIR itself, but fails to generalize to other attacks. On the contrary, DAT achieves great robustness on a broad range of attacks.

b) Larger pool without filtering. Extending the training set from 12 to 16 samples per behavior removes the benefit of filtering on Llama3-8B, which aligns with our theoretical motivation:

Llama3-8BGCG ↓GCG+ ↓PAIR ↓BoN ↓Inpainting ↓WCA ↓XSTest ↑MMLU ↑ARC-E ↑ARC-C ↑
DAT-top160.080.130.090.000.320.360.4640.6390.8130.521
DAT-random160.080.130.100.000.320.330.4760.6400.8110.523

Filtering helps at 12 samples per behavior (Table 11), where random-12 (0.6) has higher WCA than top-12 (0.35). Random-16 reaches the same robustness as top-16 without any selection on the target model. We plan to run the same comparison on Qwen2.5-14B and Gemma3-12B and further investigate the role of filtering.

c) Held-out natural harmful prompts. We use 252 held-out harmful questions from the GPT-generated portion of SALAD-Data [1], screened for content and checked for overlap against HarmBench and JailbreakBench. We submitted each question without any attack and report ASR@32, with HarmBench judge:

ASR@32 ↓No defenseCATDAT
Llama3-8B10.71%1.59%0.79%
Qwen2.5-14B13.10%1.98%0.40%
Gemma3-12B23.02%0.79%0.79%

DAT never trains on these questions and lowers ASR@32 below the undefended model on all three families, and below or equal to CAT. Robustness learned from generated prompts therefore transfers to natural harmful requests outside the training distribution.

Action 1.1: We will add the top-16 / random-16 comparison to Appendix C.7.

Action 1.2: We will add the held-out evaluation on 252 natural harmful questions as a new appendix section.

Q2: Independent judge and robustness–utility trade-off.

Can you score attacks with a judge different from the one used for filtering, and report the robustness–utility trade-off

A2: Under HarmBench classifier [2] (cais/HarmBench-Llama-2-13b-cls), the advantage of DAT persists. We rejudge table 1 with Harmbench and show below a snapshot of the comparison StrongREJECT / HarmBench:

ModelAttackCATMixATDAT
Llama3-8BInpainting0.88 / 0.700.90 / 0.710.24 / 0.07
Llama3-8BClaudini0.09 / 0.090.61 / 0.760.09 / 0.08
Llama3-8BBoN0.00 / 0.000.00 / 0.000.00 / 0.00
Qwen2.5-14BInpainting0.97 / 0.680.79 / 0.480.08 / 0.04
Qwen2.5-14BClaudini0.33 / 0.300.21 / 0.260.00 / 0.01

HarmBench judges Inpainting responses more strictly than StrongREJECT for all three defenses. The advantage of DAT over the baselines stays the same or grows. This therefore confirms that DAT does not exploit the judge used for filtering.

Action 2.1: We will add the HarmBench judge for Table 1 results in the appendix.

Q3: Adaptive and multi-turn attacks.

How does DAT hold up against adaptive attacks designed with knowledge of the defense, and against multi-turn jailbreaks

A3: We emphasize that DAT modifies only the model weights and introduces no inference-time defenses, input filters, or gradient-obfuscating components. Thus, our attack selection already contains the attacks a defense-aware attacker would choose against DAT. Every attack in our suite except Direct uses feedback from the defended model itself. GCG, GCG+ and Claudini optimize against its gradients, PAIR refines prompts on its responses, and BoN and Inpainting select the best of 1000 attack responses.

We additionally evaluate two attacker models. Indirect Harmfulness Optimization (IHO) [4] finetunes an attacker with iterative preference optimization, using a harmfulness judge as reward and only black-box access to the target. We run it against each defense separately for four optimization cycles and then sample 64 attack prompts per behavior. Jailbreak-R1 (JR1) [3] is an RL-trained attacker LLM that generates diverse jailbreak prompts. We use the released attacker policy, sample 20 attack prompts per behavior, and evaluate the same prompts on every defense. Both attacks use the first 40 JailbreakBench behaviors (Llama3-8B):

AttackJudgeNo defenseCATMixATDAT
IHOStrongREJECT0.8250.3000.2750.050
IHOHarmBench0.5500.1500.1500.025
JR1StrongREJECT0.5250.4250.1250.175
JR1HarmBench0.5750.2500.1000.100

DAT lowers JR1 ASR relative to CAT under both judges. MixAT, trained on LLM-generated persuasion prompts, reaches a similar level. JR1 provides additional results that DAT generalizes well to unseen attacks. IHO adapts its attacker to the responses of each defended model. With the same IHO budget for every defense, DAT has the lowest IHO ASR under both judges. We are currently running more attacks. However, since attack generation is computationally expensive, we might not be able to report results during the rebuttal.

Action 3.1: We will add the JR1 and IHO evaluation to the appendix.

Thank you again for the constructive feedback. We have added the top-16 versus random-16 comparison, the held-out evaluation on natural harmful questions, the HarmBench re-judging of Table 1, and two additional attacks (JR1 and IHO). We hope these revisions address your concerns and are happy to provide further clarification if needed.

[1] Lijun Li et al. SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models. Findings of ACL 2024.

[2] Mantas Mazeika et al. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML 2024.

[3] Weiyang Guo et al. Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning. 2025.

[4] Vincent Limbach et al. All you need to break LLMs are Black-Box, Adapting, Efficient, Transferable, Harmful, Applicable ... Attacks. 2026.

Dear Reviewer hwjg,

We would like to thank you once again for your detailed review and for recognizing that DAT achieves the best worst-case robustness on all three models and generalizes to unseen attack families.

In response to your review, we provided the following clarifications and additional experiments, most of which are now included in the revised paper:

  • Tested the main takeaway of Theorem 4.1 empirically, through sample scaling and through training on unfiltered diffusion samples, which matches top-score filtering at 16 samples per behavior (App. C.7).
  • Clarified the contribution beyond MixAT in the general comment: rather than swapping the prompt generator, DAT changes the training distribution itself, sampling new prompts from an approximation of q(x|y) instead of rephrasing fixed seed prompts. The controlled PAIR-DAT and MixAT comparisons show that attack-based generators do not generalize beyond their own family (App. C.2).
  • Separated distributional coverage from attack strength, as you suggested: a generator comparison at matched attack success (PAIR-DAT), a larger unfiltered pool, and a held-out evaluation on 252 natural harmful questions from SALAD-Data, where DAT achieves the lowest or equal ASR on all three models (App. C.9).
  • Re-judged Table 1 with the HarmBench classifier, where the advantage of DAT stays the same or grows.
  • Evaluated two additional attacker models, including the adaptive IHO attack optimized against each defense separately, where DAT reaches 0.050 ASR versus 0.300 for CAT and 0.275 for MixAT, as well as Jailbreak-R1 (App. C.10).
  • Compared DAT with CAT at matched XSTest compliance on all three models (App. C.8) and added a WCA–XSTest trade-off plot for all three models (Fig. 4).

As we approach the end of the discussion period, we would appreciate hearing whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining questions or provide further clarifications.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment to all Reviewers (1)

We thank all reviewers for their careful reviews. All three reviews raise two shared questions: whether the robustness advantage of DAT holds at matched benign compliance, and what DAT contributes beyond MixAT. We answer both here.

Robustness at matched compliance

hwjg (Weakness 3), 5pWQ (Weakness 2), wz9u (Weakness 3, Question 3)

Thank you for the comment. We agree that this is an important comparison. At matched XSTest compliance, DAT has a much lower WCA than CAT on all three models, and also a lower WCA/Inp. We compare CAT with DAT models trained with different retain weights and training lengths. As generating attacks is costly, we evaluate them with PAIR, GCG and Inpainting on the first 40 JailbreakBench behaviors for the rebuttal and will conduct a full evaluation for the final paper. WCA is taken over these three attacks, and WCA/Inp. over PAIR and GCG.

Table R1a. Qwen2.5-14B at matched XSTest compliance.

MethodXSTest ↑ARC-E ↑PAIR ↓GCG ↓Inpainting ↓WCA ↓WCA/Inp. ↓
CAT0.5440.8570.2500.1250.9250.9250.300
DAT, rw 0.35, 2 epochs0.6840.8580.3000.3500.8250.8500.425
DAT, rw 0.35, 3 epochs0.5280.8540.1750.1250.2500.4250.250
DAT, rw 0.85, 6 epochs0.5120.8510.1500.1000.2500.3750.200

At the same compliance level, DAT lowers the WCA from 0.925 to 0.425 and the WCA/Inp. from 0.300 to 0.250. Both methods show the same utility on ARC-E for this checkpoint (DAT: 0.854, CAT: 0.857).

On Llama3-8B, a DAT run with retain weight 0.85 from our training-run sweep exceeds the XSTest compliance of CAT, again on the first 40 JBB behaviors:

Table R1b. Llama3-8B at matched XSTest compliance.

MethodXSTest ↑PAIR ↓GCG ↓Inpainting ↓WCA ↓WCA/Inp. ↓
CAT0.4640.2500.1000.9000.9000.275
DAT, rw 0.45 (paper)0.4240.1750.0500.2250.3500.200
DAT, rw 0.85, batch 80.4920.2000.0750.2250.3500.225
DAT, rw 0.85, batch 640.4440.1000.0750.3750.4000.125

At the same compliance level, DAT lowers the WCA from 0.900 to 0.350 and the WCA/Inp. from 0.275 to 0.125. The robustness advantage of DAT therefore does not come from stronger refusal. Note that we only include three attacks here for clear comparison. With further attacks, the gap is likely to grow: JR1 and IHO break CAT on 0.425 and 0.300 of the behaviors, against 0.175 and 0.050 for DAT (response to Reviewer hwjg, A3).

On Gemma3-12B, we trained DAT with different retain weights and training lengths. All three checkpoints exceed the XSTest compliance of CAT, again on the first 40 JBB behaviors:

Table R1c. Gemma3-12B at matched XSTest compliance.

MethodXSTest ↑PAIR ↓GCG ↓Inpainting ↓WCA ↓WCA/Inp. ↓
CAT0.4880.5500.4500.9000.9000.725
DAT, rw 0.34, 5 epochs0.4960.1750.1500.0000.3000.300
DAT, rw 0.345, 6 epochs0.5080.1750.1000.0000.2500.250
DAT, rw 0.34, 6 epochs0.5120.1500.1750.0000.2750.275

At higher compliance than CAT, all three DAT checkpoints lower the WCA from 0.900 to at most 0.300, and the WCA/Inp. from 0.725 to at most 0.300.

Action: We add Tables R1a–c to the updated paper, and will extend the Inpainting-only Fig. 3 with a WCA–XSTest Pareto plot for all three models in Fig. 4.

[1] Yair Carmon et al. Unlabeled Data Improves Adversarial Robustness. NeurIPS 2019.

[2] Sven Gowal et al. Improving Robustness using Generated Data. NeurIPS 2021.

[3] Zekai Wang et al. Better Diffusion Models Further Improve Adversarial Training. ICML 2023.

AuthorsOpenReview ↗

General Comment to all Reviewers (2)

Novelty

hwjg (Weakness 2), 5pWQ (Weakness 1), wz9u (Weakness 2, Question 2)

We thank the reviewers for this question and will make the distinction to MixAT more explicit in the paper. MixAT and DAT address different parts of adversarial training: MixAT combines discrete and continuous attacks to create a stronger adversary (the inner loop), whereas we view DAT as a data augmentation method to better cover the training distribution (the outer loop). This follows an established line of work in vision, where adversarial training was substantially improved by augmenting the training data (Carmon et al., 2019; Gowal et al., 2021; Wang et al., 2023).

We argue that this line of work will get even more important in LLMs. In computer vision, perturbations are bounded by a clear ℓp\ell_p threat model. In contrast, the space of discrete adversarial prompts in LLMs is unbounded, and a defender cannot train on every possible attack. Therefore, contrary to the premise that the choice of prompt generator is an incremental detail, we argue that how to design the training distribution under a finite budget so that it generalizes to unseen attacks is a fundamental research question in LLM adversarial training. DAT provides a principled answer: we derive the target distribution for the outer loop (q(x∣y)q(x \mid y), Eq. 1), establish criteria for approximating it (Theorem 4.1, Sec. 4.2), and show that a single source meeting these criteria generalizes to diverse unseen attacks where attack-mixing fails. The resulting procedure is deliberately simple: diffusion inpainting followed by CAT. The non-trivial part is identifying which distribution to sample from, as the results for MixAT and PAIR-DAT below show. We highlight four points:

1. A different training distribution, not a different generator MixAT trains on a local perturbation set around each fixed seed prompt, RPAP(x)+B2(0,ϵ)\mathcal{R}_{\text{PAP}}(x) + \mathcal{B}^2(0,\epsilon) (MixAT Eq. 8), i.e., rephrasings of xx. DAT instead targets the harmful population risk q~(x,y)=q~(y) q(x∣y)\tilde{q}(x,y) = \tilde{q}(y)\,q(x \mid y) (Eq. 1): it discards xx and samples new, data-specific prompts from a learned approximation of q(x∣y)q(x \mid y). Theorem 4.1 shows that the gap to population robustness is bounded by the Total Variation distance to q(x∣y)q(x \mid y), which motivates our criteria for the training distribution: conditional inversion, data-specificity, and diversity (Sec. 4.2). Diffusion inpainting meets these criteria (Sec. 4.3, Fig. 2). Attack generators such as PAP or PAIR are not designed to approximate q(x∣y)q(x \mid y), and models trained on them remain vulnerable to attacks outside their own family (points 2 and 3).

2. Attack-based training like MixAT has clear failure cases If the choice of prompt generator within CAT were trivial, MixAT would be broadly robust. Instead, training on heuristic rephrasings (PAP) only defends against PAP-like rewrites:

  • In our evaluation (Table 1), MixAT suffers 94% Worst-Case ASR (WCA) on Llama3-8B (92% even excluding Inpainting), failing against GCG (31%), Claudini (61%), and GCG+ (78%), whereas DAT achieves 34% WCA (17% ex-Inpainting).
  • Even in MixAT’s own paper (Dékány et al., Table 1 & Sec. 4.3), PAP+CAT fails against GCG (22.5% ASR on Llama3-8B). To patch this, the authors had to add online GCG (MixAT+GCG) at 8x higher compute (Table 5). DAT still outperforms MixAT+GCG by a wide margin using a single offline source (34% vs. 77% WCA on Llama3-8B; 18% vs. 75% on Qwen2.5-14B; Table 8).

3. Controlled Ablation on the data generator (App. C.2, Table 7) To directly test whether any strong prompt generator works inside our exact pipeline, we trained PAIR-DAT (holding target responses, StrongREJECT filtering, 1,200 sample budget, and modified CAT constant, only swapping diffusion for PAIR). Like MixAT, PAIR-DAT overfits to its own attack family (0% PAIR ASR) while not providing robustness to unseen attacks like GCG (37%) and GCG+ (70% vs. 6% for DAT) on all 100 JailbreakBench behaviors, extending the 40 behaviors of Table 7. This provides additional evidence that finding the right data distribution is non-trivial.

4. Value of a Simple, Scalable Single-Source Method Prior work concludes that broad robustness requires mixing multiple, potentially expensive discrete attacks during training (e.g., MixAT+GCG). Showing that a single diffusion generator can replace complex multi-attack ensembles while cutting WCA by more than half at 1/8th the training cost is a considerable practical and methodological advance.

More broadly, we view DAT as an important step showing that systematically studying the properties of the outer-loop training distribution (such as transferability, diversity, and harmfulness filtering; Fig. 2, Sec. 6.5) is a critical, underexplored research direction for improving robustness generalization in LLM adversarial training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.