acceptodds
Under review as a conference paper at ICLR 2027

Improving Fuzzy Robustness in LLM Safety with Automated Red Teaming

Abstract

AI safety and alignment must robustly stay ahead of capabilities to mitigate significant risks. In this paper, we propose a scalable approach to generate adversarial safety data and show that reinforcement learning (RL) training improves the robustness of large language models (LLMs) on safety in biology and cybersecurity. Motivated by robust optimization with computationally bounded adversaries, we scale the test-time compute of reasoning models through a feedback loop harness that allows them to iteratively propose attacks, query target LLMs, and refine candidates using safety judge feedback. We show that reasoning models without specialized red-teaming training can already achieve high attack success rates (ASR) against frontier LLMs, while trained attackers perform better still. To mitigate reward hacking from fuzzy safety feedback, we use the worst@n attacker objective, which requires success across repeated evaluations, and show that increasing yields jailbreaks that are more reliable and transferable to a held-out defender LLM. Finally, we apply automated red-teaming to generate a large set of adversarial tasks in cybersecurity and biology and incorporate these robustified tasks into RL post-training. In a controlled comparison, training with robustified tasks improves safety's robustness on held-out fixed jailbreaks from 45% to 68% and on adaptive online attacks from 71% to 92%. In sum, training on robustified tasks generally improves fuzzy robustness to new tasks and attacks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI Safety, Fuzzy robustness, Automatic red teaming harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wADhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how automated red teaming can be used not only to discover jailbreaks but also to generate adversarial training data that improves the safety robustness of large language models. The proposed framework uses a reasoning-model attacker that repeatedly modifies a seed task, queries one or more defenders, and receives judge feedback before proposing the next attack. The main methodological addition is a worst@n criterion that accepts an attack only when repeated defender samples and safety judgments all indicate failure, with the goal of reducing selection on noisy safety feedback. The resulting attacks are then incorporated into a reinforcement-learning post-training mixture, and the authors evaluate whether the robustness gains generalize to held-out fixed jailbreaks, persona-based adaptive attacks, and separately RL-trained attackers. The empirical story is broad and generally coherent, with evidence that stricter search objectives produce more reliable attacks and that robustness training improves performance against several attack families.

Critical strengths

The strongest aspect of the paper is that it evaluates robustness under substantially more pressure than a standard static jailbreak benchmark. The progression from independently resampled attacks, to held-out attacker models, to multi-turn persona attacks, and finally to an RL-trained attacker is a useful experimental design because it explicitly tests whether gains survive increasingly adaptive adversaries. The high-level motivation is also easy to follow: the distinction between broad safety coverage and robustness within that coverage gives the paper a clear conceptual narrative from the first page onward.

Critical weaknesses

My main concern is that the paper does not convincingly establish that the reported gains come from the proposed notion of fuzzy robustness without adding more difficult safety examples to the training mixture. The authors themselves note that robustification can be interpreted as simply extending the curriculum with harder prompts, and the training experiment does not include an equally sized hard-example control, non-adversarial paraphrase augmentation, ordinary jailbreak augmentation, or another data-selection strategy matched for training budget and prompt difficulty. The feedback-loop generation procedure or worst@n filtering is not shown as necessary for the downstream robustness gain.

Questions for authors

Could the authors provide evidence that the feedback-loop search and worst@n filtering are necessary for the final robustness gains?

2 comments

Dear Reviewer,

Thank you for your review and for taking a close look at our paper.

Is the proposed robustification necessary for the final robustness gains?

We do not attempt to claim necessity; we only claim that our approach is sufficient for improving fuzzy robustness. Our main contribution is showing that the proposed robustification is simple, effective and can be readily used by frontier model developers to improve the safety and robustness of their models which is a timely and pressing issue with the rapid rise in capabilities. With regards to ablating the training set with distributions of difficult prompts, we remark that it is difficult to source harmful prompts in cyber and biology due to their sensitive nature. As such we view our robustification harness as a scalable approach of synthesizing difficult prompts. We leave ablating variations of attacker styles as promising future work.

Thank you again for taking the time to review our work.

I thank the authors for their response to my question. My main concern remains that the paper does not show the gains come from the proposed method, and the rebuttal does not provide new evidence on this. I will keep my score.

Reviewer_3JMtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an automated red-teaming (ART) framework to improve the adversarial robustness of LLM safety training when safety objectives are fuzzy and safety judgments are noisy. The framework uses reasoning-based attacker models to iteratively generate jailbreaks, query target defenders, and refine attacks based on feedback. The resulting adversarial examples are incorporated into RL post-training to improve defender robustness against unseen attacks.

Critical strengths
  1. The authors develop an iterative ART framework that enables reasoning models to discover effective jailbreaks through repeated interaction with defender models and safety judges.

  2. The paper introduces a stricter attack success criterion (or the worst@n attacker objective). Experiments demonstrate that increasing n improves jailbreak reliability under independent evaluation and transferability to held-out defenders.

  3. The authors demonstrate that training an attacker specifically for red teaming improves attack efficiency and effectiveness. Compared with a general reasoning attacker, the trained attacker increases attack success rates from 37.5% to 68.3% in cybersecurity and from approximately 85% to 91% in biology.

  4. Incorporating ART-generated adversarial examples into RL post-training improves safety against held-out fixed jailbreaks (45% → 68%) and adaptive multi-turn attacks (71% → 92%).

Critical weaknesses
  1. The paper builds upon two well-established research directions: iterative adversarial attack generation and worst-case robustness optimization. Feedback-driven adversarial search and iterative attacker–defender training have been extensively explored in prior work, while adversarial training based on challenging or worst-case examples has long been studied in robust optimization [1,2,3]. The proposed framework combines these established principles, with worst@n serving as a reliability-aware criterion for attack selection. Specifically, the paper does not compare the proposed approach with alternative worst-case or risk-sensitive objectives under matched computational budgets, making it difficult to assess whether it represents a meaningful methodological advance beyond existing adversarial robustness techniques.

  2. Although the robustness-trained defender achieves substantial improvements against fixed jailbreaks and persona-based adaptive attacks, Figure 7 reveals that these gains diminish as the RL-trained attacker becomes stronger, with the defender's safety performance eventually deteriorating to approximately the baseline level. This finding raises concerns about whether the proposed method meaningfully improves adversarial robustness or merely increases the optimization effort required to discover successful attacks. More importantly, because the defender is already trained on adversarial examples generated through iterative red teaming, its continued vulnerability to further attack optimization suggests that the training procedure may improve resistance primarily within the explored training distribution rather than generalize to stronger adversarial strategies. The paper lacks a systematic analysis of attack-budget-dependent robustness, including whether the newly discovered successful attacks differ meaningfully from those encountered during training. Consequently, the evidence does not sufficiently establish that the proposed approach provides sustained robustness improvements under stronger adaptive adversaries, which is central to the paper's claims.

  3. In addition, the paper does not systematically characterize the diversity of generated attacks or examine the trade-off between attack reliability and coverage of the adversarial space. Consequently, it remains unclear whether optimizing worst@n produces a sufficiently representative adversarial training distribution or encourages concentration on a limited set of exploitable vulnerabilities.

References:

[1] Madry et al. "Towards Deep Learning Models Resistant to Adversarial Attacks." International Conference on Learning Representations, 2018.

[2] Zhang et al. “Theoretically Principled Trade-off between Robustness and Accuracy.” Proceedings of the 36th International Conference on Machine Learning, 2019

[3] Sagawa et al. "Distributionally Robust Neural Networks." International Conference on Learning Representations, 2020.

Questions for authors

Given that the paper primarily focuses on closed-source commercial LLMs, it would be valuable to compare the proposed adversarial robustness training approach with alternative safety mechanisms, such as external safety classifiers [4, 5] and representation-level safety probes [6]. These approaches can provide an additional layer of protection without modifying the underlying model and may preserve its general capabilities more effectively. Could the authors clarify the advantages of incorporating adversarial examples directly into RL post-training rather than deploying lightweight and robust safety mechanisms?

References

[4] Inan et al. "Llama guard: LLM-based input-output safeguard for human-AI conversations." 2023

[5] Sharma et al. "Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming." 2025

[6] Cunningham et al. "Constitutional classifiers++: Efficient production-grade defenses against universal jailbreaks." 2026

2 comments

Dear Reviewer,

Thank you for your thoughtful review.

Does the proposed method improve adversarial robustness or merely increase the optimization effort required to discover successful attacks?

We do not view these two notions as fundamentally different: if more optimization or effort is required to successfully break defender B than A, then defender B is intuitively more robust than A. We formalize this notion in our definition of “attacker with bounded computation” in Equation (2), where robustness is not guaranteed for all possible attackers but only bounded attackers. With deep learning models, we believe that the right objective for robustness is the amount of effort it takes an attacker to find a jailbreak, because it is hard to prove definitively that a deep neural network does not have jailbreaks and empirically it has been possible to find adversarial examples given high enough attacker optimization pressure or effort.

Does the robustness gain generalize to other attack strategies?

In our evaluations, we see many encouraging signs of generalization to new attacker distributions, including in Figure 6 which shows transfer to both (a) static jailbreaks from an unseen attacker model, and (b) persona-based adaptive attackers. We provide an additional detailed breakdown of budget-dependent robustness for the persona-results in Appendix B.2, where we show that across all out-of-distribution 12 personas, the robustness-trained defender exhibits greater worst-case robustness than the baseline.

Why train the defender if we can use a classifier or probe?

We agree that classifiers and probes are quite effective at preventing jailbreaks, but they are not perfectly robust. Therefore, we believe an ideal safety system should have defense-in-depth, meaning that safety and robustness is optimized at every layer of the inference stack including the model itself. Model robustness is also critical for open-weight models which can be served without external safeguards. We’ll add this discussion and the relevant citations you provided to the final paper.

Thank you again for taking the time to review our work.

Thank you for the clarifications. While I acknowledge that requiring greater computational effort to discover successful attacks constitutes a meaningful improvement in robustness, my main concerns remain insufficiently addressed. First, the proposed framework largely builds on established adversarial search and robustness optimization techniques, without comparisons against alternative objectives under matched computational budgets to substantiate its methodological novelty. Second, although Figure 6 provides evidence of generalization to held-out attacks, Figure 7 shows that robustness gains substantially diminish under stronger attacker optimization, raising questions about how far these improvements generalize beyond the explored adversarial distribution. Finally, the rebuttal does not address the diversity of generated attacks or the potential trade-off between attack reliability and adversarial coverage. Overall, the rebuttal does not provide sufficient new evidence to resolve these concerns, and I therefore maintain my original assessment.

Reviewer_ay5yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the ART harness for automated red teaming, where a attacker model propose attacks, and refined attacks based on judge's feedback. The authors shows that the LLM models without specialized training can already achieves high ASR against gpt model families. Furthermore, this paper proposes to use worst@n attacker objective, which gives more reliable and transferable attacks to a defender model. As last, a post training on top of the collected attacks helps the model's robustness against jailbreak.

Critical strengths
  1. The ART harness design makes sense and is well motivated. It improves on top of PAIR by incorporating feedback from the judge directly. ART preserves the attacker's KV cache across feedback steps rather than re-prompting strengthen the attacker.
  2. The observation that fuzzy safety judge make iterative attack search prone to stopping on false positives is valuable.
  3. The robustness training is evaluated against different level of attacker pressure, which gives comprehensive analysis on the experiment result.
Critical weaknesses
  1. The proposed harness ART is incremental to PAIR, which is a single stream iterative red-teaming framework. Applying feedback from a judge to iterate the attack attempt is not novel enough. And the mentioned worst@n has been proposed and utilized in other related paper as a metric for robust attacks.
  2. There is no ablation and experiment on the safety judge. Not enough information is given on how the authors setup the judge. Without this, the results are not grounded and are not comparable with other approaches. The judge's agreement with human experts is never mentioned or measured in the paper. A judge from the same model family could be biased toward certain attacker which further bias the entire results.
  3. The result for the two RL post-training runs are marginal and not significant. RL post-training could be very high variance, the current result is not sufficient to show the advantage on the safety metrics.
Questions for authors
  1. How does ART compare to PAIR or TAP at matched token budget?
  2. How the judge's agreement with human experts in your experiments?
1 comment

Dear Reviewer,

Thanks for taking the time to provide valuable suggestions and questions!

How does ART compare to other red-teaming approaches?

In the appendix, we provide additional results comparing with Best-of-N (BoN) Jailbreaking, which tries independent variations of the seed prompt. The table below reports attack success rates (ASR) from Figure 12a, where we see that ART is significantly more efficient that BoN jailbreaking.

Token budgetART worst@2ART worst@5BoN worst@2BoN worst@5
2 million27.1%1.5%0.0%0.0%
5 million55.0%21.2%0.4%0.0%
10 million70.1%45.3%1.0%0.0%
25 million88.7%75.7%1.2%0.0%
50 million96.9%88.7%1.2%0.0%

We see ART as an agentic adaptation and simplification of PAIR and TAP, which were proposed before agents could sustain long trajectories. In ART, we simply give the attacker a budget and let it decide whether to refine the prompt or try something else. We expect custom scaffolds to become less useful as models become more capable. One more benefit of ART is that it is one long agentic rollout, so inference is more efficient than multiple API calls.

Agreement between the safety judge and human experts

Our safety judge grades responses against a rubric, which can be biased due to the rubric or how the judge LLM interprets the rubric. We worked with subject-matter experts to calibrate our safety judge, and found that worst@n judges reduced grader hacking and had better agreement. Throughout the process, we identified several ways where attacks could fool the judge and fixed those issues as discussed in Section 4. We are unable to give further details on the exact judge configuration which is proprietary.

Thank you again for your thoughtful review.

Comments on the paper

1 comment

Dear reviewers, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.