- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds on Self-RedTeam, where one policy plays both attacker and defender in online safety self-play. BoundaryPlay changes how the training requests are made: the attacker writes a harmful and a benign request together around the same goal and framing, the pair is scored jointly (harmful compliance, benign refusal and word overlap), and observed defender weaknesses are fed back to the attacker. The defender objective stays per-sample. The main experiment uses an abliterated Llama-3.1-8B with six seeds and four arms matched in defender candidate exposure. BoundaryPlay improves the JailbreakBench joint score by 6.58 points over independent generation and 4.45 over the Self-RedTeam reproduction, and lowers HarmBench ASR by 5.69 points, with a 3.41-point loss on FalseReject. On Qwen2.5 and on ordinary Llama all arms lose benign compliance, and a single-seed change of the defender reward recovers part of the IFEval loss.
Critical strengths
The comparison is carefully controlled. The four arms are matched in generated fraction and defender exposure, there are six training seeds with paired tests, and the paper also reports the results that go against it (alternative judges, full-channel generation, PAIR, the aligned models). This makes me trust the numbers that are reported. The observation on ordinary Llama that the refusal-correctness reward on the benign side is linked to much of the over-refusal is useful for people working on this kind of training.
Critical weaknesses
-
The benefit is shown only for the abliterated model, and there the model stays very unsafe after training (ASR 62.84% on JBB and 79.95% on HarmBench). On the two normally aligned models the picture is negative. On Qwen all arms lose a lot of benign compliance, and on ordinary Llama BoundaryPlay is the worst of the four arms on every benign panel (JBB Comply 45 vs 89 for Base, FalseReject 11.9 vs 51.1, IFEval 61.0 vs 75.2). The reward change recovers IFEval, but FalseReject only goes back to 17.35, and this is one seed. So I am not sure in which realistic setting one would want to use this method.
-
It is hard to attribute the gain to the counterfactual construction, which is the stated contribution. BoundaryPlay vs Homo-Split is +1.52 with CI [-5.91, 8.94], the held-out audit finds that only 36-45% of the pairs differ in a single safety-relevant aspect, and PairAcc shows no difference between the arms. The gain is also fragile: with Claude or Qwen as the judge the interval crosses zero, the HarmBench difference reverses under full-channel generation (+8.54), in the full-channel adaptive test BoundaryPlay is worse than the Self-RedTeam reproduction (73.87% vs 65.54% ASR, Appendix C), and under PAIR in chat the ASR is above 90% either way.
Questions for authors
- Is there a setting with an ordinarily aligned starting model where BoundaryPlay (with the reward change) is better than the independent baselines on both the harmful and the benign side, over more than one seed?
- Training and the main evaluation both use WildGuard. How much of the JBB gain remains when the harmful-side label comes from a judge that was not used in training? Table 5 suggests it shrinks.
- Why is the abliterated model the main testbed?
Thank you for the clarifications and additional experiments, which have addressed some of my concerns. Since I have not yet had the opportunity to review the revised paper, I will keep my score at 2 for now. We can discuss further when you revise your paper. I would also appreciate further clarification on the safety–helpfulness advantage over independent generation and the remaining robustness limitations under full-channel and adaptive attacks.