acceptodds
Under review as a conference paper at ICLR 2027

Larger Is Not Always Better: Perturbation Budgets in Adversarial Training for LLMs

Abstract

Latent-space adversarial training offers a scalable way to harden large language models against jailbreaks by optimizing adversarial perturbations in continuous representation space. A natural design intuition, inherited from computer vision, is that giving the training adversary a larger perturbation budget should expose the model to stronger attacks and thereby improve robustness. In learned LLM representation spaces, however, perturbation scales lack a natural interpretation, and their effects may differ across robustness and utility evaluations. We test how far this intuition holds through controlled perturbation-budget sweeps across configurations, evaluating the resulting models on jailbreak attacks, general capabilities, and benign compliance. We find no universal monotonic robustness trend: increasing the budget can improve robustness against some attacks, leave others unchanged, or even make robustness worse, depending on the attack and training configuration. The utility cost is equally uneven. Benign compliance can collapse while capability benchmarks remain comparatively stable, whereas in other settings capability degrades substantially with much smaller changes in benign compliance. Moreover, sufficiently large budgets can produce strictly worse robustness–utility trade-offs than smaller ones. To understand why budget alone is an incomplete description of training, we measure the perturbations actually realized by the inner optimizer. Larger configured budgets do produce larger perturbations, but with decreasing budget utilization, while the perturbations remain local to their original token embeddings. We further ask whether the dominant behavioral failure mode, over-refusal, can be repaired. Targeted supervision substantially improves behavior on the supervised over-refusal distribution, yet does not reliably recover broader benign compliance and consistently harms adversarial robustness. Overall, our results highlight both the promise and the challenges of training against stronger latent-space adversaries: larger perturbation budgets can improve robustness, but future methods must use that additional budget effectively, generalize robustness across attack families, remain stable across training runs, and avoid behavioral costs such as over-refusal.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, adversarial training, latent-space adversarial training, jailbreak robustness, robustness–utility trade-offs, over-refusal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RmCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study examines how the perturbation radius in adversarial training in LLM representation space (CAT and LAT) affects robustness against jailbreak attacks and model utility. A larger perturbation budget does not necessarily improve robustness; depending on the attack method and training setup, it can make robustness worse. With CAT, responses to harmless requests can deteriorate substantially even when performance on general tasks remains relatively intact. Additional training to reduce over-refusal helped on held-out questions from the same dataset, but its benefits did not consistently extend to an independent set of harmless questions, and it reduced jailbreak robustness. The authors also show that the actual perturbations do not grow as much as the specified radius would suggest, and that CAT perturbations remain close to the original token embeddings.

Critical strengths

The authors systematically vary the perturbation budget across multiple training setups and evaluate robustness against different jailbreak attacks, performance on general tasks, and responses to harmless requests. Although the novelty of these findings raises concerns discussed under Weaknesses, the results are novel at least in their treatment of representation space and LLM response behavior.

Critical weaknesses

My main concern is that the finding that large perturbations can sometimes degrade adversarial training is not novel. For example, the following studies identified this issue and proposed ways to improve adversarial training:

  • Q.-Z. Cai et al. Curriculum Adversarial Training. IJCAI, 2018.
  • Y. Wang et al. On the Convergence and Robustness of Adversarial Training. ICML, 2019.
  • J. Zhang et al. Attacks Which Do Not Kill Training Make Adversarial Learning Stronger. ICML, 2020.
  • A. Shaeiri et al. Towards Deep Learning Models Resistant to Large Perturbations. arXiv, 2020.
  • C. Sitawarin et al. SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing. ACM Workshop, 2021.

The paper’s central finding has already been reported in a substantial body of work, despite differences in the setting, such as representation space and LLM responses. The authors identify phenomena specific to LLMs and CAT/LAT, including how LLMs respond to harmless questions and what perturbations in representation space mean geometrically in token space. While these are new contributions, I find it difficult to regard them as sufficient for acceptance. Beyond the extensive experiments, a new CAT or LAT method informed by the analysis could strengthen the paper’s contribution.

Questions for authors

I would like the authors to clarify how this study makes a sufficient contribution relative to the prior work discussed under Weaknesses.

Reviewer_fYJPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how perturbation budgets affect LLM adversarial training through CAT sweeps and complementary LAT experiments. It reports attack- and configuration-dependent robustness trends, separates capability loss from benign-compliance loss, and analyzes realized perturbations. It also examines whether targeted over-refusal supervision can recover benign compliance without sacrificing robustness.

Critical strengths
  1. The distinction between task capability and benign compliance is useful. The results show that capability benchmarks can remain relatively stable while CAT substantially reduces responses to harmless requests, highlighting a practical limitation of capability-only evaluation (Section 4.2).

  2. The main experiments provide useful coverage and transparent reporting. The CAT and LAT sweeps evaluate multiple configurations with three training seeds per setting, while the CAT diagnostics explicitly distinguish configured budgets from realized perturbations (Sections 3 and 5).

Critical weaknesses

1.Method design: the perturbation budget is confounded with the optimizer step size. With zero initialization, 10 updates, and an update norm of 0.1ε, the iterates remain within the constraint even without projection, so varying ε effectively changes the absolute step size (Table 1; Eq. 4). The K = 100 diagnostic lacks robustness and utility evaluation, leaving unclear whether the reported reversals persist under more fully optimized inner attacks (Appendix C.3).

2.Related work: the connection to earlier budget studies is not sufficiently developed. Section 2.2 acknowledges that CAT/CAPO already examines radius-dependent robustness–utility trade-offs and that CAT over-refusal is known. The broader evaluation adds useful evidence, but the paper provides limited explanation of how its non-monotonic trends relate to earlier findings obtained under different training objectives and evaluation settings.

3.Experiments and analysis: the evidence for recovery beyond OR-Bench is not strong enough. Harmless contains only 40 prompts, while XSTest uses a prefix-based refusal detector with acknowledged disagreement with an auxiliary judge (Table 3; Appendix A.2). Most replacement experiments also use a single training seed despite substantial baseline variability, limiting confidence in the conclusion about failure to recover broader benign compliance (Appendix D).

Questions for authors

1.Can you report robustness, utility, and achieved inner attack loss for the existing K = 100 checkpoints, and test whether the robustness reversals persist when radius and step size are varied independently?

2.Can you compare your main findings directly with the prior CAT/CAPO and MixAT budget analyses, clarifying which conclusions extend earlier work and how differences in training or evaluation may explain the different radius trends?

3.Can you repeat the key OR-Bench replacement comparisons across training seeds and re-evaluate XSTest with a validated refusal judge to establish whether the cross-dataset conclusions remain consistent?

Reviewer_euNWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the perturbation budget affects latent-space adversarial training for LLM jailbreak robustness. The authors perform controlled radius sweeps for Continuous Adversarial Training (CAT) across several training configurations, complement them with Latent Adversarial Training (LAT) sweeps, and evaluate the resulting checkpoints using several jailbreak attacks, utility benchmarks, and compliance metrics. They additionally measure the perturbations actually realized by the CAT inner optimizer and investigate whether targeted supervision can repair the observed over-refusal behavior. The main finding is that increasing the configured perturbation radius does not induce a consistent monotonic improvement in robustness: the response depends on the attack and training configuration, while different components of utility can degrade very differently.

Critical strengths

S1: The paper provides a broad and carefully controlled empirical characterization of an important hyperparameter in latent-space adversarial training. The perturbation budget is central to CAT/LAT-style methods, yet its effect on downstream jailbreak robustness has not been characterized as systematically as it is here. The paper considers multiple radii, training configurations, attack families, utility benchmarks, and compliance measures, and further checks whether the qualitative behavior extends from CAT to LAT. I found the resulting empirical picture useful, especially because the robustness ordering changes across attacks and configurations rather than following a simple monotonic trend.

S2: The distinction between configured radius, realized perturbation magnitude, and different notions of utility is useful and practically relevant. Measuring the realized CAT perturbation distance DD and utilization D/ϵD/\epsilon gives a more informative picture of what the finite inner optimizer actually does than reporting the configured radius alone. Similarly, separating conventional utility benchmarks from compliance behavior reveals an important failure mode: in several CAT settings, benign compliance deteriorates substantially while MMLU/ARC scores remain comparatively stable up to intermediate radii. This is a useful warning against treating a single score as an adequate measure of the utility cost of adversarial training.

Critical weaknesses

W1: The main empirical phenomenon is only moderately novel, and the paper characterizes the non-monotonicity more clearly than it explains it. The broad conclusion that increasing the perturbation budget need not uniformly improve downstream robustness or the robustness–utility trade-off is useful, but it is also conceptually plausible from prior adversarial-training literature. Classical work already establishes robustness–accuracy trade-offs [1,2], robustness trade-offs across threat models [3], and important failure modes of weak finite-step inner adversaries [4,5]. In particular, Andriushchenko and Flammarion [5] show that improving the quality of the inner adversary can prevent catastrophic overfitting and enable fast adversarial training at larger perturbation budgets. In the LLM setting, prior CAT/MixAT work has also already studied perturbation-related hyperparameters [6,7].

The novelty relative to the original CAT paper [6] is particularly worth clarifying. The present paper states that the prior CAPO analysis varies the radius jointly with the IPO coefficient. However, [6] also reports a dedicated ϵ\epsilon ablation in their Figure 3(b), separately from the β\beta ablation, where ϵ\epsilon is swept from 0.01250.0125 to 0.10.1. The present work is clearly much broader, as it adds multiple configurations, attack families, behavioral utility, training seeds, and realized-perturbation diagnostics, but I think the positioning should acknowledge more explicitly that a controlled radius ablation itself is not new.

More importantly, the paper does not establish whether a larger configured radius actually produces a stronger optimized training adversary. The diagnostics show that larger radii produce larger realized CAT perturbation norms, but perturbation magnitude is not equivalent to adversarial effectiveness. This leaves two substantially different explanations unresolved: (i) the optimized inner adversary becomes genuinely stronger as ϵ\epsilon increases, yet transfer to downstream attack families is non-monotonic; or (ii) the finite inner optimization becomes increasingly inadequate as the feasible region grows, so a larger configured budget does not correspond to a proportionally stronger solved inner problem. Appendix C.4 strengthens this concern: the lowest-loss iterate is at step 10 in the median case and the mean selected iteration is always above 9.29.2, showing that the loss-selected solution is frequently attained only at the final inner step and raising the possibility that K=10K=10 is insufficient for convergence. Reporting the achieved inner objective across radii would therefore substantially deepen the central result and could change my assessment.

W2: The perturbation-radius effect is strongly coupled to the inner optimization procedure, and some of the empirical comparisons have limited discriminative power. I appreciate that the CAT experiments intentionally keep the update magnitude fixed relative to the radius, and the authors explicitly state that this normalization is intended to make the sweep controlled rather than to claim a superior optimizer. Nevertheless, this means that changing ϵ\epsilon necessarily also changes the absolute inner update size, so the effect of the feasible region cannot be interpreted independently of the finite optimizer.

The LAT results make this interaction particularly visible. Both LAT conventions actually scale the inner learning rate linearly with the radius: the Original-LR sweep uses LR=ϵ/6000\mathrm{LR}=\epsilon/6000, whereas the 10%-step sweep uses LR=ϵ/640\mathrm{LR}=\epsilon/640. Under the paper's first-step AdamW approximation, this corresponds to initial update norms of approximately 0.0107ϵ0.0107\epsilon and 0.1ϵ0.1\epsilon, respectively, i.e., about a 9.4×9.4\times difference in relative first-step magnitude. Yet the Soft-Prompt trends are qualitatively different between the two sweeps. This is one of the most interesting results in the paper, but it suggests that the relevant object is the interaction between radius and inner optimization, rather than radius alone. Unlike CAT, the paper does not report realized LAT perturbation norms/utilization or achieved inner-objective values, which makes the two LAT sweeps difficult to interpret mechanistically.

There is also a noticeable discrepancy between the released LAT checkpoint and the reproduced Original-LR setting at the same nominal ϵ=6\epsilon=6, LR=10−3\mathrm{LR}=10^{-3} setting. For example, MMLU is 58.2358.23 for the released checkpoint versus 48.05±2.2548.05\pm2.25 for the reproduction, ARC-C is 42.6642.66 versus 28.10±0.4428.10\pm0.44, and Soft-Prompt ASR is 62.5862.58 versus 38.15±7.7938.15\pm7.79. This does not invalidate the within-sweep comparisons, but it makes the strong capability degradation at the low end of the Original-LR sweep harder to interpret as a generic LAT radius effect without understanding the source of the reproduction gap.

Finally, the evaluation is broader in attack count than in effective discriminative power. AutoDAN rapidly reaches a near-zero floor for many trained checkpoints, while Inpainting remains close to its ceiling and is a transfer-style rather than per-checkpoint adaptive evaluation. Much of the informative radius-dependent robustness comparison therefore comes from GCG and Soft-Prompts. The main sweeps also use only three training seeds, with some metrics exhibiting very large between-seed variation, so I would interpret fine-grained dominance/orderings between nearby settings more cautiously than the large qualitative trends.

Overall, my current rating is driven primarily by explanatory depth. The study is careful and informative, but at present I view it mainly as a thorough characterization of a phenomenon that is partly expected from prior work. Evidence showing how the achieved inner adversarial objective changes with radius, and whether better-converged inner optimization changes the downstream trend, would make the central result substantially more informative and could move my assessment toward acceptance.

Additional minor comments

  • The unmodified ϵ=0\epsilon=0 model is excluded from the main dominance analysis, although the appendix notes that it becomes non-dominated in almost all robustness–utility pairings when included. I understand the motivation for comparing positive-radius settings, but showing this practical baseline more prominently would make the trade-off analysis easier to interpret.

  • The LAT XSTest unsafe-refusal values are striking: the base Llama 3 model has 91.5%91.5\% unsafe refusal, whereas the released LAT checkpoint and all LAT sweep checkpoints are around 2.82.8–7.0%7.0\%, despite several of them improving substantially on GCG/AutoDAN. Since the paper already identifies limitations in the prefix-based XSTest refusal classifier, I would avoid interpreting this directly as unsafe compliance, but the discrepancy seems large enough to deserve discussion or comparison with the auxiliary judge.

  • The targeted over-refusal experiments are substantially less controlled than the main sweeps because many replacement conditions are single-seed. In addition, the loss-weight experiment fixes αu=0.9405\alpha_u=0.9405 while varying αOR\alpha_{\mathrm{OR}}, so the total utility-side weight αu+αOR\alpha_u+\alpha_{\mathrm{OR}} is not held fixed and reaches 1.89251.8925 at the largest setting. This does not explain away the robustness degradation, which is already visible at low OR weights, but it limits the causal interpretation of the high-weight part of the sweep.

References

[1] Tsipras et al., Robustness May Be at Odds with Accuracy, ICLR 2019.

[2] Zhang et al., Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.

[3] Tramèr et al., Adversarial Training and Robustness for Multiple Perturbations, NeurIPS 2019.

[4] Wong et al., Fast Is Better Than Free: Revisiting Adversarial Training, ICLR 2020.

[5] Andriushchenko et al., Understanding and Improving Fast Adversarial Training, NeurIPS 2020.

[6] Xhonneux et al., Efficient Adversarial Training in LLMs with Continuous Attacks, NeurIPS 2024.

[7] Dékány et al., MixAT: Combining Continuous and Discrete Adversarial Training for LLMs, NeurIPS 2025.

Questions for authors

Q1: How does the achieved CAT inner objective vary with ϵ\epsilon? Since CAT already selects the lowest-loss inner-loop iterate, could you report the corresponding inner adversarial loss (or improvement relative to the unperturbed input) across the radius sweep? Is the achieved objective monotonic in ϵ\epsilon?

Q2: What happens if the better-optimized K=100K=100 Zephyr checkpoints are evaluated for robustness and utility? Appendix C.3 shows that the larger inner-loop budget reaches approximately full utilization, but the corresponding checkpoints are not evaluated downstream. Even a limited comparison would help distinguish a radius effect from an inner-optimization effect.

Q3: What perturbation magnitudes and inner-objective values are actually realized by the two LAT sweeps? Since both use LR∝ϵ\mathrm{LR}\propto\epsilon but differ by roughly 9.4×9.4\times in relative first-step size and produce opposite Soft-Prompt trends, reporting analogous LAT diagnostics to those used for CAT would make this result much easier to interpret.

Q4: How do you distinguish the present novelty from the dedicated ϵ\epsilon ablation already reported in the original CAT paper [6]? The current related-work discussion emphasizes a jointly varied ϵ\epsilon/β\beta analysis, but [6] also includes a separate ϵ\epsilon ablation in Figure 3(b). I agree that the present study is substantially broader; I would mainly like the claimed delta relative to that experiment to be stated more precisely.

Q5: What explains the gap between the released LAT checkpoint and the reproduced Original-LR ϵ=6\epsilon=6 checkpoints? The discrepancy is large on both capability and Soft-Prompt robustness. Is this expected from training-seed variation, an implementation/training difference, or another factor? This matters for interpreting the unusual low-ϵ\epsilon behavior in the Original-LR sweep.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.