- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper applies Policy-Space Response Oracles (PSRO) to LLM safety training, combining equilibrium-weighted populations of frozen attackers and defenders, freshly initialized LoRA adapters, and efficient payoff estimation to address stagnation and forgetting in self-play.
Critical strengths
-
The paper addresses a meaningful limitation of latest-opponent self-play. Preserving historical policies offers a reasonable way to retain complementary strategies and reduce forgetting.
-
Comparisons with latest-opponent training, uniform population sampling, and warm-start initialization help explain the contributions of the main design choices.
-
Racing evaluation and rollout caching make population-based training more practical. The reported reward-hacking cases also provide useful insights into the challenges of extended adversarial training.
Critical weaknesses
-
The connection between population diversity and overcoming stagnation needs stronger evidence. The paper motivates its approach through strategy collapse and performance plateaus, but the evidence connecting population retention to sustained exploration remains limited. Preserving historical policies does not necessarily ensure that the effective training distribution remains diverse, particularly when equilibrium weights concentrate on a few members. Moreover, the population-level behavioral analysis uses an earlier reward configuration rather than the configuration producing the main results. Tracking complementary attack coverage, newly discovered vulnerabilities, and forgetting across generations under the final configuration would better establish how population growth addresses the stated problem.
-
Model-family coverage and comparison with strong co-evolutionary methods are limited. The main experiments use an abliterated Llama-3.1-8B backbone, leaving it unclear whether the findings generalize to other model families and models with their original safety alignment intact. The Qwen results taken from prior work do not substitute for directly training PSRO on Qwen. In addition, MAGIC already performs separate attacker–defender optimization but is not included as an experimental baseline. Direct experiments across model families and a comparison with MAGIC under consistent evaluation conditions would strengthen the evidence for the additional benefit of population-based training.
-
The advantage over existing experience-replay approaches is not established. SSP [1], which this paper cites, already retains and revisits historical failure cases through reflective experience replay. However, it is not included in the experimental comparison. This is a relevant alternative because replay can preserve difficult experiences without retaining a population of historical models. A comparison would help determine whether the gains require policy-level population retention and equilibrium weighting, or whether existing sample-level memory mechanisms can achieve comparable results more efficiently.
-
Reward hacking and dependence on the training judge remain unresolved concerns. WildGuard supplies the training rewards and evaluates most of the main safety benchmarks. Consequently, improvements may partly reflect adaptation to the same evaluator’s preferences or blind spots, a concern reinforced by the reward hacking observed during development. The additional StrongREJECT and SORRY-Bench graders provide useful independent evidence, but do not directly test agreement on the same outputs used in the main evaluation. Moreover, Appendix C acknowledges that the label-based fixes do not cover substitution of one harmful task with another. The current evidence therefore does not establish that the revised training objective adequately controls reward hacking. Independent reassessment of the same outputs, including disagreement analysis and checks of attack fidelity to the original task, would strengthen the evidence.
[1] Wang et al. Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay (https://aclanthology.org/2026.findings-acl.933/). Findings of ACL, 2026.
Questions for authors
-
Does population growth sustain meaningful strategy diversity and continued discovery of new vulnerabilities?
-
Do the benefits generalize across model families and remain competitive with MAGIC?
-
What additional benefit does policy-population retention provide over existing experience replay?
-
To what extent does the revised objective mitigate reward hacking rather than shift exploitation toward other weaknesses of the reward and judge?
5 comments
Q4. How does PSRO compare with MAGIC?
We compared both methods using the same evaluation pipeline, first verifying that it reproduces the WildGuardTest attack-success rates reported in MAGIC’s Table 1 for the released checkpoint. We then trained MAGIC using its official code on the same abliterated Llama-3.1-8B base as PSRO, with two runs per method. As shown in Table R3, PSRO achieves approximately 10 percentage points higher safety, while MAGIC retains higher helpfulness. The safety gap is consistent across all four judges when reassessing the same outputs (Table R5).
Table R3. Safety and helpfulness under the same evaluation protocol. Results are mean ± SD over two runs.
| Method | Safety | Helpfulness |
|---|---|---|
| No Training | .356 | .936 |
| PSRO (ours) | .955 ± .010 | .811 ± .047 |
| MAGIC | .850 ± .007 | .920 ± .071 |
Q5. Do the findings generalize to other model families and to models whose safety alignment is intact?
We have now trained PSRO directly on an abliterated Qwen3-8B (Table R4). As with abliterated Llama-3.1-8B, the starting model rarely refuses harmful requests. PSRO raises its safety from .429 to .983 within four generations, at a larger helpfulness cost than on Llama (.966 to .740). We did not run the baselines on Qwen, so this shows that the method transfers to a second model family, not that its advantage over other methods does. We have not trained PSRO from a model whose original safety alignment is intact. We expect two difficulties in that setting: an already well-aligned defender leaves limited headroom for the safety benchmarks we use (the instruction-tuned Llama-3.1-8B reaches a safety above .93 in our evaluation), and an attacker initialized from an aligned model may struggle to produce effective attacks, which would make successful-attack rewards sparse. We will state this as a limitation.
Table R4. PSRO on two model families, under the same evaluation protocol.
| Model | Safety | Helpfulness |
|---|---|---|
| Abliterated Llama-3.1-8B (no training) | .356 | .936 |
| + PSRO (generation 5, two seeds) | .955 ± .010 | .811 ± .047 |
| Abliterated Qwen3-8B (no training) | .429 | .966 |
| + PSRO (generation 4) | .983 | .740 |
Q6. Why is SSP not included as a baseline?
We view SSP as complementary to PSRO, which is why we did not include it as a baseline. Both retain past experience, but at different levels: SSP selects which samples a single model revisits, replaying recorded failures with UCB weighting; PSRO selects which opponents each new policy trains against, retaining policies with Nash equilibrium weighting. A retained policy can generate fresh attacks as defenders change, whereas recorded cases remain fixed. SSP’s replay could therefore be incorporated into each PSRO oracle, allowing new policies to revisit their own failures. We will clarify this distinction in the related work and discuss the comparison and combination as future work.
Q7. Could using WildGuard for both training and evaluation bias the results, and do independent judges agree?
We re-judged the same defender outputs on WildJailbreak and HarmBench (3,197 prompts) with three judges independent of training: the StrongREJECT fine-tuned grader, the HarmBench classifier, and Gemini-3.8-Flash with the official StrongREJECT rubric (Table R5). The ranking of the four methods is the same under every judge, and PSRO has the lowest harmful rate under all four.
Table R5. Harmful-response rate of the generation-5 defender on the same 3,197 outputs. PSRO, fictitious self-play and MAGIC: mean over two seeds.
| Method | WildGuard (training judge) | StrongREJECT | HarmBench classifier | Gemini-3.8-Flash |
|---|---|---|---|---|
| PSRO | .031 | .029 | .028 | .016 |
| Fictitious self-play | .062 | .054 | .049 | .026 |
| MAGIC | .091 | .193 | .149 | .068 |
| Naive self-play | .417 | .230 | .278 | .092 |
Q8. Does the revised objective mitigate reward hacking, or shift it elsewhere, and do attacks stay faithful to the original task?
The revised objective does not fully control reward hacking: it addresses label drift but not topic substitution, where the attacker replaces a harmful task with a different one. To check attack fidelity, we used Gemini-3.8-Flash, at evaluation time only, to label whether each attacker’s rewrite of 300 harmful seed prompts switches to a different harmful task (Table R6). Substitution peaks at A2–A3 and is lower at A4–A5, although it remains in 10.0% of A5’s rewrites. It inflates the attacker’s reward but does not mislead the defender: a substituted prompt is still a harmful request, so the defender is still trained to refuse it. Our defender results come from fixed external benchmarks whose prompts never pass through the attacker, and hold under the independent judges of Table R5. Fully preventing substitution would require a fidelity-aware reward, which we leave to future work.
Table R6. Share of attacks that switch to a different harmful task (300 seeds per attacker).
| Attacker | A1 | A2 | A3 | A4 | A5 |
|---|---|---|---|---|---|
| Different harmful task | 3.0% | 33.3% | 33.3% | 2.3% | 10.0% |
We thank the reviewer for the detailed comments. Below, we address each point with additional experiments and clarifications on population diversity, evaluation, and the scope of our results.
By ``diversity," we mean the phenomenon illustrated in Figure 2(a), which we will describe as partial convergence: each training run converges to only a subset of effective strategies. Retaining a population preserves the union of these subsets, helping overcome the stagnation of an individual policy. We do not reward diversity explicitly. We attribute it to two likely sources: different training data can lead different runs to learn different strategies, and different opponents can push each new policy toward different weaknesses.
We agree that the weights concentrate. PSRO aims to converge to a Nash equilibrium rather than maintain a broad mixture indefinitely. As shown in Table 3 of the main paper, the equilibrium mixes several members early and concentrates on a single member later. This concentration does not increase exploitability against the retained population: the highest attacker payoff decreases from −0.28 to −0.70, indicating that the concentrated defender successfully counters every attacker in the final population.
We conducted this analysis under the final configuration, on the attackers and defenders of the reported run and 300 harmful seed prompts.
Complementary coverage. To test whether the finding in Figure 2 (a) carries over to the final reward, we conducted a small-scale analysis with two attackers trained under the final configuration. Against the same defender, they succeed on 42.0% and 25.3% of the seeds, respectively, with an overlap of only 10.0%. Their combined coverage is 57.3%, or 1.37× that of the stronger attacker alone. Each attacker thus finds vulnerabilities that the other misses, consistent with Figure 2.
Vulnerability discovery across generations. We labeled each attacker’s rewrites of the same seeds with tactic families, following the protocol of Appendix F, and also characterized each rewrite by its opening (its first four words; Table R2). On the same seed, consecutive attackers rarely use the same tactics (mean Jaccard overlap 0.14–0.32), and the leading tactic changes over the first three generations, from persona to research and fiction and then to direct requests. A4 and A5 return to persona-led attacks, although 82% and 74% of their opening templates are not used by any earlier attacker.
Table R2. Attackers of the reported run on the same 300 harmful seeds. Primary tactic: the tactic family a rewrite opens with (keyword rules as in Appendix F; “direct request” if none matches). Values are percentages except the last row.