acceptodds
Under review as a conference paper at ICLR 2027

BoundaryPlay: Improving Online Safety Self-Play via Counterfactual Request Construction

Abstract

In a weak-safety setting, online safety self-play adapts training requests to a changing defender, but how harmful and benign requests are constructed remains underexplored. Independent generation leaves their relationship unconstrained. We introduce BoundaryPlay, a counterfactual request-construction protocol: the attacker uses a seed goal and shared framing, aiming to vary safety intent between harmful and legitimate benign requests while preserving unrelated content. The pair is scored jointly for inducing harmful compliance and benign refusal, and the attacker receives feedback on observed defender weaknesses; the per-sample defender objective is unchanged. Against an independent-generation control matched in generated fraction and defender candidate exposure, BoundaryPlay improves the JailbreakBench joint safety–compliance score on Llama-3.1-8B-Instruct-abliterated by 6.58 points (4.45 over a Self-RedTeam RL-only reproduction), with gains positive in all six seeds and significant in a post-hoc Holm sensitivity analysis, and reduces HarmBench harmful responses by 5.69 points against the same control, at a 3.41-point cost in FalseReject compliance. On initially aligned Llama, we identify the defender's plus-side refusal-correctness reward as a modifiable contributor to instruction-following loss: removing this component while retaining counterfactual construction recovers 11.83 points of Instruction-Following Evaluation (IFEval) strict prompt accuracy and improves all four benign compliance panels, with few additional harmful responses on the evaluated harmful-request panels. BoundaryPlay thus contributes both a request-construction protocol and a targeted reward-design intervention that mitigates instruction-following loss in the tested single-seed setting.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language model safety, online safety self-play, adversarial training, counterfactual generation, red teaming, safety-helpfulness trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Uqt3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on Self-RedTeam, where one policy plays both attacker and defender in online safety self-play. BoundaryPlay changes how the training requests are made: the attacker writes a harmful and a benign request together around the same goal and framing, the pair is scored jointly (harmful compliance, benign refusal and word overlap), and observed defender weaknesses are fed back to the attacker. The defender objective stays per-sample. The main experiment uses an abliterated Llama-3.1-8B with six seeds and four arms matched in defender candidate exposure. BoundaryPlay improves the JailbreakBench joint score by 6.58 points over independent generation and 4.45 over the Self-RedTeam reproduction, and lowers HarmBench ASR by 5.69 points, with a 3.41-point loss on FalseReject. On Qwen2.5 and on ordinary Llama all arms lose benign compliance, and a single-seed change of the defender reward recovers part of the IFEval loss.

Critical strengths

The comparison is carefully controlled. The four arms are matched in generated fraction and defender exposure, there are six training seeds with paired tests, and the paper also reports the results that go against it (alternative judges, full-channel generation, PAIR, the aligned models). This makes me trust the numbers that are reported. The observation on ordinary Llama that the refusal-correctness reward on the benign side is linked to much of the over-refusal is useful for people working on this kind of training.

Critical weaknesses
  1. The benefit is shown only for the abliterated model, and there the model stays very unsafe after training (ASR 62.84% on JBB and 79.95% on HarmBench). On the two normally aligned models the picture is negative. On Qwen all arms lose a lot of benign compliance, and on ordinary Llama BoundaryPlay is the worst of the four arms on every benign panel (JBB Comply 45 vs 89 for Base, FalseReject 11.9 vs 51.1, IFEval 61.0 vs 75.2). The reward change recovers IFEval, but FalseReject only goes back to 17.35, and this is one seed. So I am not sure in which realistic setting one would want to use this method.

  2. It is hard to attribute the gain to the counterfactual construction, which is the stated contribution. BoundaryPlay vs Homo-Split is +1.52 with CI [-5.91, 8.94], the held-out audit finds that only 36-45% of the pairs differ in a single safety-relevant aspect, and PairAcc shows no difference between the arms. The gain is also fragile: with Claude or Qwen as the judge the interval crosses zero, the HarmBench difference reverses under full-channel generation (+8.54), in the full-channel adaptive test BoundaryPlay is worse than the Self-RedTeam reproduction (73.87% vs 65.54% ASR, Appendix C), and under PAIR in chat the ASR is above 90% either way.

Questions for authors
  1. Is there a setting with an ordinarily aligned starting model where BoundaryPlay (with the reward change) is better than the independent baselines on both the harmful and the benign side, over more than one seed?
  2. Training and the main evaluation both use WildGuard. How much of the JBB gain remains when the harmful-side label comes from a judge that was not used in training? Table 5 suggests it shrinks.
  3. Why is the abliterated model the main testbed?
Reviewer_Xb2zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BoundaryPlay jointly generates harmful and benign requests that share similar topics and narrative framing, but differ in safety intent, and uses these pairs for online safety self-play. It also introduces a joint attacker reward and weakness feedback. The defender is still trained on individual examples. The paper compares the complete protocol starting from a weakly safe model, and separately studies losses in benign capability, plus a reward correction, when the starting point is an already aligned model.

Critical strengths

Training on harmful and benign requests that share a topic has a clear motivation: the model should learn to distinguish intent, rather than refusing simply because a request contains sensitive words. On the weakly safe Llama starting point, the complete protocol improves the JBB joint score over independent generation by 6.58 percentage points, and the difference is positive in all six seeds. This is the strongest positive evidence in the paper, and it is worth reporting.

Critical weaknesses

A sufficiently convincing improvement in the joint safety and benign-capability trade-off has not been established. For initially aligned Llama, BoundaryPlay is below all three training controls on all four groups of benign results, and IFEval also drops substantially. The reward correction recovers some of that performance, but it is evaluated with only one seed. Even after recovery, IFEval remains below independent generation, and benign compliance on FalseReject is still low. The relative gain from a weakly safe starting point is real, but it does not show that this is a more useful safety-training method once the model is already aligned.

The method’s most distinctive element, counterfactual request construction, is also not isolated. In Table 1, joint generation, joint reward, and weakness feedback all change together, while Homo-Split changes both target coverage and input organization. The results therefore support the complete protocol, without showing how much the paired construction itself contributes. The paper also moves frequently among protocol-level gains, the pairing mechanism, and the later reward correction, which makes the central contribution less focused.

Overall assessment: I lean toward rejection. The complete protocol does have gains from a particular starting point, but neither the distinctive value of the request construction nor a more useful safety-capability trade-off is sufficiently established to support acceptance.

Questions for authors

Which existing control most directly supports shared-topic request construction, rather than joint reward or feedback, as the source of the main improvement? Please specify the mechanistic interpretation that the current results can actually support.

After reward correction, how does benign capability compare with independent generation at the same safety level? Are there existing results that show a better trade-off, rather than only a partial recovery from losses introduced by the proposed method?

Reviewer_cK8kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BoundaryPlay, an online safety self-play method that generates paired harmful and benign requests from a shared seed goal. Its joint attacker reward targets both harmful compliance and benign over-refusal, while the defender is trained on each request independently. The experiments show improvements in the safety–compliance trade-off, though the contribution of counterfactual request construction is not fully isolated.

Critical strengths
  1. The joint attacker reward targets both harmful compliance and benign over-refusal. It combines these signals plus content-word overlap, penalizes invalid labels, and thus captures harmful-request failures and benign over-refusals—not just attack success.

  2. Results are seed-robust: on weak-safety abliterated Llama-3.1-8B-Instruct, BoundaryPlay improves the JailbreakBench joint safety–compliance score by 6.58 points across all six seeds.

  3. The defender objective remains per-sample for a clear implementation boundary: requests are handled independently with no pairwise defender loss; pair structure comes from request construction and the joint attacker reward, not extra defender optimization.

Critical weaknesses

The experimental evidence does not isolate the effect of counterfactual request construction. BoundaryPlay differs from IndepGen in multiple ways (shared framing, joint attacker reward, weakness feedback, and attacker training distribution), so matching only generated-request fraction and defender candidate exposure cannot attribute gains to counterfactual requests alone. Homo-Split adds a control, but BoundaryPlay’s advantage over it remains unclear. Additional ablations holding reward, feedback, and goal coverage constant are needed to disentangle each component’s contribution.

Questions for authors

See weaknesses

Comments on the paper

1 comment

Thank you for the clarifications and additional experiments, which have addressed some of my concerns. Since I have not yet had the opportunity to review the revised paper, I will keep my score at 2 for now. We can discuss further when you revise your paper. I would also appreciate further clarification on the safety–helpfulness advantage over independent generation and the remaining robustness limitations under full-channel and adaptive attacks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.