W1: The main empirical phenomenon is only moderately novel, and the paper characterizes the non-monotonicity more clearly than it explains it. The broad conclusion that increasing the perturbation budget need not uniformly improve downstream robustness or the robustness–utility trade-off is useful, but it is also conceptually plausible from prior adversarial-training literature. Classical work already establishes robustness–accuracy trade-offs [1,2], robustness trade-offs across threat models [3], and important failure modes of weak finite-step inner adversaries [4,5]. In particular, Andriushchenko and Flammarion [5] show that improving the quality of the inner adversary can prevent catastrophic overfitting and enable fast adversarial training at larger perturbation budgets. In the LLM setting, prior CAT/MixAT work has also already studied perturbation-related hyperparameters [6,7].
The novelty relative to the original CAT paper [6] is particularly worth clarifying. The present paper states that the prior CAPO analysis varies the radius jointly with the IPO coefficient. However, [6] also reports a dedicated ϵ ablation in their Figure 3(b), separately from the β ablation, where ϵ is swept from 0.0125 to 0.1. The present work is clearly much broader, as it adds multiple configurations, attack families, behavioral utility, training seeds, and realized-perturbation diagnostics, but I think the positioning should acknowledge more explicitly that a controlled radius ablation itself is not new.
More importantly, the paper does not establish whether a larger configured radius actually produces a stronger optimized training adversary. The diagnostics show that larger radii produce larger realized CAT perturbation norms, but perturbation magnitude is not equivalent to adversarial effectiveness. This leaves two substantially different explanations unresolved: (i) the optimized inner adversary becomes genuinely stronger as ϵ increases, yet transfer to downstream attack families is non-monotonic; or (ii) the finite inner optimization becomes increasingly inadequate as the feasible region grows, so a larger configured budget does not correspond to a proportionally stronger solved inner problem. Appendix C.4 strengthens this concern: the lowest-loss iterate is at step 10 in the median case and the mean selected iteration is always above 9.2, showing that the loss-selected solution is frequently attained only at the final inner step and raising the possibility that K=10 is insufficient for convergence. Reporting the achieved inner objective across radii would therefore substantially deepen the central result and could change my assessment.
W2: The perturbation-radius effect is strongly coupled to the inner optimization procedure, and some of the empirical comparisons have limited discriminative power. I appreciate that the CAT experiments intentionally keep the update magnitude fixed relative to the radius, and the authors explicitly state that this normalization is intended to make the sweep controlled rather than to claim a superior optimizer. Nevertheless, this means that changing ϵ necessarily also changes the absolute inner update size, so the effect of the feasible region cannot be interpreted independently of the finite optimizer.
The LAT results make this interaction particularly visible. Both LAT conventions actually scale the inner learning rate linearly with the radius: the Original-LR sweep uses LR=ϵ/6000, whereas the 10%-step sweep uses LR=ϵ/640. Under the paper's first-step AdamW approximation, this corresponds to initial update norms of approximately 0.0107ϵ and 0.1ϵ, respectively, i.e., about a 9.4× difference in relative first-step magnitude. Yet the Soft-Prompt trends are qualitatively different between the two sweeps. This is one of the most interesting results in the paper, but it suggests that the relevant object is the interaction between radius and inner optimization, rather than radius alone. Unlike CAT, the paper does not report realized LAT perturbation norms/utilization or achieved inner-objective values, which makes the two LAT sweeps difficult to interpret mechanistically.
There is also a noticeable discrepancy between the released LAT checkpoint and the reproduced Original-LR setting at the same nominal ϵ=6, LR=10−3 setting. For example, MMLU is 58.23 for the released checkpoint versus 48.05±2.25 for the reproduction, ARC-C is 42.66 versus 28.10±0.44, and Soft-Prompt ASR is 62.58 versus 38.15±7.79. This does not invalidate the within-sweep comparisons, but it makes the strong capability degradation at the low end of the Original-LR sweep harder to interpret as a generic LAT radius effect without understanding the source of the reproduction gap.
Finally, the evaluation is broader in attack count than in effective discriminative power. AutoDAN rapidly reaches a near-zero floor for many trained checkpoints, while Inpainting remains close to its ceiling and is a transfer-style rather than per-checkpoint adaptive evaluation. Much of the informative radius-dependent robustness comparison therefore comes from GCG and Soft-Prompts. The main sweeps also use only three training seeds, with some metrics exhibiting very large between-seed variation, so I would interpret fine-grained dominance/orderings between nearby settings more cautiously than the large qualitative trends.
Overall, my current rating is driven primarily by explanatory depth. The study is careful and informative, but at present I view it mainly as a thorough characterization of a phenomenon that is partly expected from prior work. Evidence showing how the achieved inner adversarial objective changes with radius, and whether better-converged inner optimization changes the downstream trend, would make the central result substantially more informative and could move my assessment toward acceptance.
Additional minor comments
-
The unmodified ϵ=0 model is excluded from the main dominance analysis, although the appendix notes that it becomes non-dominated in almost all robustness–utility pairings when included. I understand the motivation for comparing positive-radius settings, but showing this practical baseline more prominently would make the trade-off analysis easier to interpret.
-
The LAT XSTest unsafe-refusal values are striking: the base Llama 3 model has 91.5% unsafe refusal, whereas the released LAT checkpoint and all LAT sweep checkpoints are around 2.8–7.0%, despite several of them improving substantially on GCG/AutoDAN. Since the paper already identifies limitations in the prefix-based XSTest refusal classifier, I would avoid interpreting this directly as unsafe compliance, but the discrepancy seems large enough to deserve discussion or comparison with the auxiliary judge.
-
The targeted over-refusal experiments are substantially less controlled than the main sweeps because many replacement conditions are single-seed. In addition, the loss-weight experiment fixes αu=0.9405 while varying αOR, so the total utility-side weight αu+αOR is not held fixed and reaches 1.8925 at the largest setting. This does not explain away the robustness degradation, which is already visible at low OR weights, but it limits the causal interpretation of the high-weight part of the sweep.
References
[1] Tsipras et al., Robustness May Be at Odds with Accuracy, ICLR 2019.
[2] Zhang et al., Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.
[3] Tramèr et al., Adversarial Training and Robustness for Multiple Perturbations, NeurIPS 2019.
[4] Wong et al., Fast Is Better Than Free: Revisiting Adversarial Training, ICLR 2020.
[5] Andriushchenko et al., Understanding and Improving Fast Adversarial Training, NeurIPS 2020.
[6] Xhonneux et al., Efficient Adversarial Training in LLMs with Continuous Attacks, NeurIPS 2024.
[7] Dékány et al., MixAT: Combining Continuous and Discrete Adversarial Training for LLMs, NeurIPS 2025.