- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how automated red teaming can be used not only to discover jailbreaks but also to generate adversarial training data that improves the safety robustness of large language models. The proposed framework uses a reasoning-model attacker that repeatedly modifies a seed task, queries one or more defenders, and receives judge feedback before proposing the next attack. The main methodological addition is a worst@n criterion that accepts an attack only when repeated defender samples and safety judgments all indicate failure, with the goal of reducing selection on noisy safety feedback. The resulting attacks are then incorporated into a reinforcement-learning post-training mixture, and the authors evaluate whether the robustness gains generalize to held-out fixed jailbreaks, persona-based adaptive attacks, and separately RL-trained attackers. The empirical story is broad and generally coherent, with evidence that stricter search objectives produce more reliable attacks and that robustness training improves performance against several attack families.
Critical strengths
The strongest aspect of the paper is that it evaluates robustness under substantially more pressure than a standard static jailbreak benchmark. The progression from independently resampled attacks, to held-out attacker models, to multi-turn persona attacks, and finally to an RL-trained attacker is a useful experimental design because it explicitly tests whether gains survive increasingly adaptive adversaries. The high-level motivation is also easy to follow: the distinction between broad safety coverage and robustness within that coverage gives the paper a clear conceptual narrative from the first page onward.
Critical weaknesses
My main concern is that the paper does not convincingly establish that the reported gains come from the proposed notion of fuzzy robustness without adding more difficult safety examples to the training mixture. The authors themselves note that robustification can be interpreted as simply extending the curriculum with harder prompts, and the training experiment does not include an equally sized hard-example control, non-adversarial paraphrase augmentation, ordinary jailbreak augmentation, or another data-selection strategy matched for training budget and prompt difficulty. The feedback-loop generation procedure or worst@n filtering is not shown as necessary for the downstream robustness gain.
Questions for authors
Could the authors provide evidence that the feedback-loop search and worst@n filtering are necessary for the final robustness gains?
2 comments
I thank the authors for their response to my question. My main concern remains that the paper does not show the gains come from the proposed method, and the rebuttal does not provide new evidence on this. I will keep my score.
Dear Reviewer,
Thank you for your review and for taking a close look at our paper.
Is the proposed robustification necessary for the final robustness gains?
We do not attempt to claim necessity; we only claim that our approach is sufficient for improving fuzzy robustness. Our main contribution is showing that the proposed robustification is simple, effective and can be readily used by frontier model developers to improve the safety and robustness of their models which is a timely and pressing issue with the rapid rise in capabilities. With regards to ablating the training set with distributions of difficult prompts, we remark that it is difficult to source harmful prompts in cyber and biology due to their sensitive nature. As such we view our robustification harness as a scalable approach of synthesizing difficult prompts. We leave ablating variations of attacker styles as promising future work.
Thank you again for taking the time to review our work.