acceptodds
Under review as a conference paper at ICLR 2027

Population-Based Self-Play for LLM Safety Training

Abstract

While self-play promises unbounded synthetic data for LLM post-training, standard single-policy methods quickly plateau. Training against only the latest opponent causes policies to cycle between behaviors and collapse onto narrow strategies. We address this by applying population-based self-play via Policy-Space Response Oracles to attacker–defender safety training. In each generation, a new policy is initialized from the base model and trained against the Nash equilibrium of the entire opposing population. To make population payoff matrices scalable at 8B parameters, we introduce a racing evaluation budget and a rollout cache. On standard safety benchmarks, our defender achieves 96.2% average safety, outperforming baseline self-play methods (at most 86.4%) at helpfulness comparable to the other self-play methods and with little loss of general capability. Ablations confirm that each element is necessary: training against single opponents induces cycling, uniform weighting degrades safety, and warm-starting triggers policy collapse. Finally, we show that strong populations expose reward specification flaws, enabling targeted fixes to the reward function.

Primary area:
reinforcement learning
Keywords:
self-play, jailbreaking, multi-agent reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_foExOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies Policy-Space Response Oracles (PSRO) to LLM safety training, combining equilibrium-weighted populations of frozen attackers and defenders, freshly initialized LoRA adapters, and efficient payoff estimation to address stagnation and forgetting in self-play.

Critical strengths
  1. The paper addresses a meaningful limitation of latest-opponent self-play. Preserving historical policies offers a reasonable way to retain complementary strategies and reduce forgetting.

  2. Comparisons with latest-opponent training, uniform population sampling, and warm-start initialization help explain the contributions of the main design choices.

  3. Racing evaluation and rollout caching make population-based training more practical. The reported reward-hacking cases also provide useful insights into the challenges of extended adversarial training.

Critical weaknesses
  1. The connection between population diversity and overcoming stagnation needs stronger evidence. The paper motivates its approach through strategy collapse and performance plateaus, but the evidence connecting population retention to sustained exploration remains limited. Preserving historical policies does not necessarily ensure that the effective training distribution remains diverse, particularly when equilibrium weights concentrate on a few members. Moreover, the population-level behavioral analysis uses an earlier reward configuration rather than the configuration producing the main results. Tracking complementary attack coverage, newly discovered vulnerabilities, and forgetting across generations under the final configuration would better establish how population growth addresses the stated problem.

  2. Model-family coverage and comparison with strong co-evolutionary methods are limited. The main experiments use an abliterated Llama-3.1-8B backbone, leaving it unclear whether the findings generalize to other model families and models with their original safety alignment intact. The Qwen results taken from prior work do not substitute for directly training PSRO on Qwen. In addition, MAGIC already performs separate attacker–defender optimization but is not included as an experimental baseline. Direct experiments across model families and a comparison with MAGIC under consistent evaluation conditions would strengthen the evidence for the additional benefit of population-based training.

  3. The advantage over existing experience-replay approaches is not established. SSP [1], which this paper cites, already retains and revisits historical failure cases through reflective experience replay. However, it is not included in the experimental comparison. This is a relevant alternative because replay can preserve difficult experiences without retaining a population of historical models. A comparison would help determine whether the gains require policy-level population retention and equilibrium weighting, or whether existing sample-level memory mechanisms can achieve comparable results more efficiently.

  4. Reward hacking and dependence on the training judge remain unresolved concerns. WildGuard supplies the training rewards and evaluates most of the main safety benchmarks. Consequently, improvements may partly reflect adaptation to the same evaluator’s preferences or blind spots, a concern reinforced by the reward hacking observed during development. The additional StrongREJECT and SORRY-Bench graders provide useful independent evidence, but do not directly test agreement on the same outputs used in the main evaluation. Moreover, Appendix C acknowledges that the label-based fixes do not cover substitution of one harmful task with another. The current evidence therefore does not establish that the revised training objective adequately controls reward hacking. Independent reassessment of the same outputs, including disagreement analysis and checks of attack fidelity to the original task, would strengthen the evidence.

[1] Wang et al. Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replay (https://aclanthology.org/2026.findings-acl.933/). Findings of ACL, 2026.

Questions for authors
  1. Does population growth sustain meaningful strategy diversity and continued discovery of new vulnerabilities?

  2. Do the benefits generalize across model families and remain competitive with MAGIC?

  3. What additional benefit does policy-population retention provide over existing experience replay?

  4. To what extent does the revised objective mitigate reward hacking rather than shift exploitation toward other weaknesses of the reward and judge?

5 comments

We thank the reviewer for the detailed comments. Below, we address each point with additional experiments and clarifications on population diversity, evaluation, and the scope of our results.

Q1. How does population diversity help overcome training stagnation?

By ``diversity," we mean the phenomenon illustrated in Figure 2(a), which we will describe as partial convergence: each training run converges to only a subset of effective strategies. Retaining a population preserves the union of these subsets, helping overcome the stagnation of an individual policy. We do not reward diversity explicitly. We attribute it to two likely sources: different training data can lead different runs to learn different strategies, and different opponents can push each new policy toward different weaknesses.

Q2. Does equilibrium weight concentration undermine training diversity?

We agree that the weights concentrate. PSRO aims to converge to a Nash equilibrium rather than maintain a broad mixture indefinitely. As shown in Table 3 of the main paper, the equilibrium mixes several members early and concentrates on a single member later. This concentration does not increase exploitability against the retained population: the highest attacker payoff decreases from −0.28 to −0.70, indicating that the concentrated defender successfully counters every attacker in the final population.

Q3. Under the final configuration, how does population growth affect complementary attack coverage, vulnerability discovery, and forgetting?

We conducted this analysis under the final configuration, on the attackers and defenders of the reported run and 300 harmful seed prompts.

  1. Complementary coverage. To test whether the finding in Figure 2 (a) carries over to the final reward, we conducted a small-scale analysis with two attackers trained under the final configuration. Against the same defender, they succeed on 42.0% and 25.3% of the seeds, respectively, with an overlap of only 10.0%. Their combined coverage is 57.3%, or 1.37× that of the stronger attacker alone. Each attacker thus finds vulnerabilities that the other misses, consistent with Figure 2.

  2. Vulnerability discovery across generations. We labeled each attacker’s rewrites of the same seeds with tactic families, following the protocol of Appendix F, and also characterized each rewrite by its opening (its first four words; Table R2). On the same seed, consecutive attackers rarely use the same tactics (mean Jaccard overlap 0.14–0.32), and the leading tactic changes over the first three generations, from persona to research and fiction and then to direct requests. A4 and A5 return to persona-led attacks, although 82% and 74% of their opening templates are not used by any earlier attacker.

Table R2. Attackers of the reported run on the same 300 harmful seeds. Primary tactic: the tactic family a rewrite opens with (keyword rules as in Appendix F; “direct request” if none matches). Values are percentages except the last row.

Primary tacticA1A2A3A4A5
Direct request1.323.750.06.77.7
Persona / role-play72.30.30.053.758.7
Research / academic1.025.06.35.74.7
Fiction / creative2.320.318.08.76.3
Journalism15.36.03.04.34.7
Hypothetical0.012.70.00.30.0
Authority / official2.34.710.07.09.7
Urgency / emotion3.00.09.07.34.0
Other2.37.33.76.34.3
Rewrites with a new opening100.0100.0100.082.374.3
Tactic overlap with previous attacker, same seed (Jaccard)–0.140.250.140.32
  1. Forgetting. A newer defender can become more vulnerable to an earlier attacker; for example, A1 does better against D2 than against D1 (Table 3(a)). Because the population is retained, these attackers remain in the payoff matrix, allowing the regression to be measured and incorporated into the next equilibrium computation. The worst-case attacker payoff against the retained population nevertheless decreases at every iteration (−0.28, −0.59, −0.60, −0.68, −0.70; Table 3(b)).

Q4. How does PSRO compare with MAGIC?

We compared both methods using the same evaluation pipeline, first verifying that it reproduces the WildGuardTest attack-success rates reported in MAGIC’s Table 1 for the released checkpoint. We then trained MAGIC using its official code on the same abliterated Llama-3.1-8B base as PSRO, with two runs per method. As shown in Table R3, PSRO achieves approximately 10 percentage points higher safety, while MAGIC retains higher helpfulness. The safety gap is consistent across all four judges when reassessing the same outputs (Table R5).

Table R3. Safety and helpfulness under the same evaluation protocol. Results are mean ± SD over two runs.

MethodSafetyHelpfulness
No Training.356.936
PSRO (ours).955 ± .010.811 ± .047
MAGIC.850 ± .007.920 ± .071

Q5. Do the findings generalize to other model families and to models whose safety alignment is intact?

We have now trained PSRO directly on an abliterated Qwen3-8B (Table R4). As with abliterated Llama-3.1-8B, the starting model rarely refuses harmful requests. PSRO raises its safety from .429 to .983 within four generations, at a larger helpfulness cost than on Llama (.966 to .740). We did not run the baselines on Qwen, so this shows that the method transfers to a second model family, not that its advantage over other methods does. We have not trained PSRO from a model whose original safety alignment is intact. We expect two difficulties in that setting: an already well-aligned defender leaves limited headroom for the safety benchmarks we use (the instruction-tuned Llama-3.1-8B reaches a safety above .93 in our evaluation), and an attacker initialized from an aligned model may struggle to produce effective attacks, which would make successful-attack rewards sparse. We will state this as a limitation.

Table R4. PSRO on two model families, under the same evaluation protocol.

ModelSafetyHelpfulness
Abliterated Llama-3.1-8B (no training).356.936
+ PSRO (generation 5, two seeds).955 ± .010.811 ± .047
Abliterated Qwen3-8B (no training).429.966
+ PSRO (generation 4).983.740

Q6. Why is SSP not included as a baseline?

We view SSP as complementary to PSRO, which is why we did not include it as a baseline. Both retain past experience, but at different levels: SSP selects which samples a single model revisits, replaying recorded failures with UCB weighting; PSRO selects which opponents each new policy trains against, retaining policies with Nash equilibrium weighting. A retained policy can generate fresh attacks as defenders change, whereas recorded cases remain fixed. SSP’s replay could therefore be incorporated into each PSRO oracle, allowing new policies to revisit their own failures. We will clarify this distinction in the related work and discuss the comparison and combination as future work.

Q7. Could using WildGuard for both training and evaluation bias the results, and do independent judges agree?

We re-judged the same defender outputs on WildJailbreak and HarmBench (3,197 prompts) with three judges independent of training: the StrongREJECT fine-tuned grader, the HarmBench classifier, and Gemini-3.8-Flash with the official StrongREJECT rubric (Table R5). The ranking of the four methods is the same under every judge, and PSRO has the lowest harmful rate under all four.

Table R5. Harmful-response rate of the generation-5 defender on the same 3,197 outputs. PSRO, fictitious self-play and MAGIC: mean over two seeds.

MethodWildGuard (training judge)StrongREJECTHarmBench classifierGemini-3.8-Flash
PSRO.031.029.028.016
Fictitious self-play.062.054.049.026
MAGIC.091.193.149.068
Naive self-play.417.230.278.092

Q8. Does the revised objective mitigate reward hacking, or shift it elsewhere, and do attacks stay faithful to the original task?

The revised objective does not fully control reward hacking: it addresses label drift but not topic substitution, where the attacker replaces a harmful task with a different one. To check attack fidelity, we used Gemini-3.8-Flash, at evaluation time only, to label whether each attacker’s rewrite of 300 harmful seed prompts switches to a different harmful task (Table R6). Substitution peaks at A2–A3 and is lower at A4–A5, although it remains in 10.0% of A5’s rewrites. It inflates the attacker’s reward but does not mislead the defender: a substituted prompt is still a harmful request, so the defender is still trained to refuse it. Our defender results come from fixed external benchmarks whose prompts never pass through the attacker, and hold under the independent judges of Table R5. Fully preventing substitution would require a fidelity-aware reward, which we leave to future work.

Table R6. Share of attacks that switch to a different harmful task (300 seeds per attacker).

AttackerA1A2A3A4A5
Different harmful task3.0%33.3%33.3%2.3%10.0%
Reviewer_yE3JOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates population-based self-play for improving LLM safety, addressing the strategy collapse and cycling problems observed in conventional single-policy self-play. The authors adapt Policy-Space Response Oracles (PSRO) to LLM safety training, maintaining populations of attackers and defenders and training each new policy against a Nash equilibrium mixture of the opposing population. They further introduce several techniques to reduce computational overhead, including racing evaluation and rollout caching. Experiments on Llama-3.1-8B demonstrate substantial improvements in safety over existing self-play baselines while maintaining comparable helpfulness. The paper also provides ablations investigating population diversity, training dynamics, computational efficiency, and reward hacking.

Critical strengths
  1. Clear motivation and excellent presentation. The paper is very well written, particularly the abstract, introduction, and motivation sections. It clearly identifies the failure modes of existing self-play approaches and explains how population-based training can address these limitations. Figure 2 provides particularly compelling evidence for the importance of maintaining a population.

  2. A well-motivated and principled approach. The adaptation of PSRO to LLM safety training is conceptually straightforward but well justified. The paper provides a clear connection between the game-theoretic formulation and the practical design choices, including equilibrium-based opponent sampling and fresh policy initialization.

  3. Comprehensive experimental evaluation. The experiments cover multiple safety and helpfulness benchmarks, with comparisons against several self-play baselines. The ablations are relatively thorough, examining the contributions of population training, equilibrium weighting, and initialization strategies. The generation-wise results further help explain the training dynamics.

  4. Useful practical insights into large-scale self-play. Beyond demonstrating performance improvements, the paper discusses computational efficiency and identifies reward hacking as an important limitation of stronger attacker populations. These observations provide useful insights for future work on scaling self-play training.

Critical weaknesses
  1. Limited generalization across model families. All the proposed PSRO experiments are conducted on a single model, Llama-3.1-8B. Although results from Qwen models are included for reference, the proposed approach itself is not evaluated on another model family. This makes it difficult to assess how well the findings generalize.

  2. Limited comparison with alternative safety improvement methods. The experiments primarily compare different self-play approaches. However, there are many other methods for improving model safety while maintaining helpfulness, including conventional safety fine-tuning and representation-based interventions. It remains unclear how the proposed method compares with these alternatives, especially considering its additional computational requirements.

  3. Computational scalability remains uncertain. The authors provide useful optimizations and demonstrate that payoff-matrix estimation is relatively inexpensive compared with training in their setting. However, maintaining and evaluating populations of large models remains computationally demanding. It is unclear how this trade-off scales to larger models or substantially larger populations, and whether the additional complexity is justified by the resulting performance improvements.

  4. Reliance on reward quality and remaining reward hacking issues. The paper demonstrates that stronger attacker populations can exploit weaknesses in the reward function. While the authors introduce mitigations, Appendix C indicates that harmful-to-harmful topic substitution is not fully addressed. This raises questions about the robustness of the training objective as the population becomes stronger.

Questions for authors
  1. How well does the proposed approach generalize to other model families or model sizes? Even an additional experiment on a different model family would help establish whether the observed improvements are specific to Llama-3.1-8B.

  2. How does the overall computational cost compare with other safety improvement methods, such as conventional safety fine-tuning or representation-based interventions? In particular, is the additional cost of population-based training justified by its improvements in safety and helpfulness?

  3. Section 4.4 suggests that both identified reward exploits have been addressed, whereas Appendix C notes that topic substitution remains unresolved. Could the authors clarify this discrepancy and discuss whether topic substitution still occurs in the final trained populations?

  4. Could the authors clarify the exact computation of the zero-sum payoff used in the empirical payoff matrix? In particular, how are the role-specific rewards described in Appendix B projected into the zero-sum payoff used for Nash equilibrium computation?

4 comments

We appreciate the reviewer’s careful reading and helpful suggestions, which prompted us to strengthen the experimental comparisons and clarify several aspects of the method. Below, we respond to each concern and describe the additional results and revisions.

Q1. Generalization of PSRO Across Model Families.

To assess generalization across model families, we additionally applied PSRO to an abliterated Qwen3-8B (Table R1). As with Llama-3.1-8B, the starting model rarely refuses harmful requests. PSRO improves its safety score from .429 to .983 within four generations, providing evidence that the approach extends to a second model family. The improvement comes with a larger helpfulness cost than on Llama. We did not evaluate the self-play baselines on Qwen or test other model sizes, so these results do not establish PSRO’s relative advantage on Qwen or its generalization across model sizes. We will include these results and limitations in the revision.

Table R1. PSRO on two model families, under the same evaluation protocol. Safety and Helpfulness are the averages defined in Table 1 of the paper.

ModelSafetyHelpfulness
Abliterated Llama-3.1-8B (no training).356.936
+ PSRO (generation 5, two seeds).955 ± .010.811 ± .047
Abliterated Qwen3-8B (no training).429.966
+ PSRO (generation 4).983.740

Q2. Comparison with representation-based interventions and the Computational Cost and Training Time of PSRO.

We added a representation-based baseline: we retrained Circuit Breakers (RR) [1] with its official recipe on the same abliterated Llama-3.1-8B base, without tuning, and evaluated it with the same pipeline (Table R2). RR blocks harmful content by derailing the output rather than refusing, so we also report harmful-response rates under four judges on the same prompts.

Table R2. Same base model and evaluation. Harmful-response rates on 3,197 outputs (WildJailbreak + HarmBench). PSRO, FSP: mean ± SD over two seeds.

MethodSafety ↑Harmful (WildGuard) ↓Harmful (StrongREJECT) ↓Harmful (HarmBench classifier) ↓Harmful (Gemini) ↓
PSRO.955 ± .010.031 ± .005.029 ± .001.028 ± .005.016 ± .004
Fictitious self-play.890 ± .037.062 ± .054.054 ± .019.049 ± .012.026 ± .007
Naive self-play.638.417.230.278.092
Circuit Breakers (RR).579.466.084.146.079
No training.356————

Starting from a base model without refusal behavior, PSRO achieves a safety score of .955, compared with .579 for RR (up from .356), and yields a lower harmful rate across all three judges. RR is substantially cheaper to train, approximately 1 hour on a single H100, compared with 40 hours on four H200s for PSRO, but does not achieve the same level of safety in this setting.

This cost gap is not specific to PSRO. It is shared by every self-play method, since all of them train both an attacker and a defender over many rounds rather than a single model once. Among population-based self-play methods, PSRO is the cheaper one: compared with fictitious self-play, it trains for 38% fewer steps over five generations (1,840 vs. 2,960), because its defenders converge earlier. Estimating the payoff matrix adds only about 6% to training time. We will add this comparison and the cost discussion to the revision.

[1] Zou, Andy, et al. "Improving alignment and robustness with circuit breakers." Advances in Neural Information Processing Systems 37 (2024): 83345-83373.

Q3. Clarifying the Scope of Reward Exploit Fixes and Topic Substitution in Final Populations

Thank you for catching this. Appendix C is correct: our fixes do not address the substitution of one harmful task for another, and Section 4.4 overstates them. We have corrected Section 4.4 accordingly in the revision.

To measure how often substitution occurs, we used Gemini-3.8-Flash as an LLM judge at evaluation time only; it was never used in training. For each attacker, it labels whether the rewrite of each of the 300 harmful seed prompts switches to a different harmful task (Table R3).

Table R3. Share of attacks that switch to a different harmful task (300 seeds per attacker; Gemini-3.8-Flash as judge).

AttackerA1A2A3A4A5
Different harmful task3.0%33.3%33.3%2.3%10.0%

Substitution peaks at A2–A3 and is lower at A4–A5, although it remains in 10.0% of A5’s rewrites.

The defender is not misled by substitution. A substituted prompt is still a harmful request, so the defender is still trained to refuse harmful content; substitution changes which harmful request it sees, not what it learns. Its main effect is to inflate the attacker’s reward, which can make the attacker a weaker training opponent than it appears. Our evaluation is unaffected: all defender results come from fixed external benchmarks whose prompts never pass through the attacker, and PSRO’s harmful rate stays low under four independent judges (.016–.031; see Q2). Fully preventing substitution would require a fidelity-aware reward, which we leave to future work.

Q4. Constructing Zero-Sum Payoffs from Role-Specific Rewards for Nash Equilibrium Computation.

Thank you for this question; the paper omits the formula, and we have added it to Appendix B in the revision.

Payoff matrix. The payoff matrix is zero-sum by construction. For each episode, we take the two role rewards from Appendix B without their chain-of-thought format terms, so each lies in [−2,2][−2, 2]. We rescale each to [−1,1][−1, 1] and average the attacker’s view with the negated defender’s view:

uA=(rA−rD)/4;uD=−uA.u_A = (r_A − r_D) / 4; u_D = −u_A.

The entry for each attacker–defender pair is the average uAu_A over all episodes played between them, with harmful and benign seeds sampled equally. The Nash equilibrium of this matrix is solved exactly by linear programming.

Training. Each policy is trained on the same role reward, rAr_A or rDr_D, before the zero-sum projection. On top of this, we add a penalty on the chain-of-thought format. This penalty matters only at the start of training, when policies still violate the format, and it quickly vanishes once they learn it, so it has little effect on what the policies learn. Each new policy is therefore an approximate best response to the zero-sum game above.

I thank the authors for the replies and additional results. I believe these will further strengthen the paper and will retain my score.

Thank you very much for your time, positive feedback, and support of our manuscript. We truly appreciate your valuable comments.

Comments on the paper

4 comments

We thank the reviewer for the thoughtful review and constructive feedback. We address your questions and concerns point by point below.

Q1. Can the safety and helpfulness means for PSRO vs. the strongest baseline be reported with multiple seeds?

We have now trained PSRO and fictitious self-play (FSP) with a second seed under the same configuration and evaluated all defenders using the same pipeline (Table R1).

Table R1. Generation-5 defenders. PSRO and FSP: mean ± SD over two seeds.

MethodSafetyHelpfulness
PSRO.955 ± .010.811 ± .047
Fictitious self-play.890 ± .037.844 ± .014
Self-RedTeam.808.830
Naive self-play.638.886
Abliterated Llama-3.1-8B (no training).356.936

PSRO achieves higher safety than FSP in both seeds, while FSP is slightly more helpful. Both population methods (PSRO and FSP) substantially exceed the single-policy methods (Self-RedTeam and Naive Self-Play) in our experiments. We will revise our claim accordingly: population-based training improves over latest-opponent training in our experiments, while PSRO consistently achieves higher safety at a modest helpfulness cost.

Q2. Is the WildJailbreak swing from 10% (generation 4) to 74% (generation 5) a sign of large run-to-run variability?

This swing is not run-to-run variability. It occurs between two consecutive generations of naive self-play within a single run (0.10 at generation 4, 0.74 at generation 5). It illustrates the cycling we describe: each new defender is trained only against the latest attacker while becoming vulnerable to earlier ones. Across seeds, the same benchmark is stable for PSRO. Its WildJailbreak attack-success rate at generation 5 is 0.092 and 0.084 for the two seeds (0.088 ± 0.006), so the 0.64 within-run swing in naive self-play is about 80× this seed-to-seed spread.

Q3. Do the safety evaluation and the training reward use the same or closely related judge? Can the 96.2% safety be re-checked with an independent judge … to rule out overfitting to the evaluation pipeline?

Yes. WildGuard, our training judge, also labels 8 of the 10 safety and over-refusal evaluation benchmarks in the paper. To check for judge-specific overfitting, we re-scored the same defender outputs on 3,197 prompts from WildJailbreak and HarmBench using three judges not used during training: the StrongREJECT fine-tuned grader, the HarmBench classifier, and Gemini-3.8-Flash (Table R2). PSRO has the lowest harmful-response rate under every judge, with rates ranging from .016 to .031. This consistency supports a safety improvement beyond WildGuard, although this check covers a subset of the benchmarks underlying the reported 95.5% safety score. Agreement between the independent judges is substantial.

Table R2. Harmful-response rates assigned by different judges to the same generation-5 defender outputs on 3,197 prompts. PSRO and FSP results are mean ± SD over two seeds.

MethodWildGuard (training judge) ↓\downarrowStrongREJECT$\downarrow$HarmBench classifier ↓\downarrowGemini-3.8-Flash ↓\downarrow
PSRO.031 ± .005.029 ± .001.028 ± .005.016 ± .004
Fictitious self-play.062 ± .054.054 ± .019.049 ± .012.026 ± .007
Naive self-play.417.230.278.092

We also checked for signs of defender-side judge exploitation: vacuous or off-topic responses that WildGuard nevertheless labels as compliant. On 250 benign WildJailbreak prompts, Gemini flagged 18 such non-answers for PSRO, compared with 43 for FSP and 70 for naive self-play. PSRO therefore shows the fewest instances of this potential failure mode. The reward hacking documented in the paper occurred on the attacker side.

Q4. The main results then use the patched reward, so sustained improvement is conditional on that manual step. Moreover, the ‘keeps improving’ narrative is short-horizon, with results limited to five generations

The reward was never changed during the main run. We used a separate eight-generation test run to find the reward flaws and patched them before the main run started. All five generations reported in the paper therefore use the same fixed reward, with no manual step between generations. Within this run, each iteration’s Nash defender is harder to break than the previous one, even for attackers it never trained against. We play each iteration’s Nash defender against every attacker in the final population, including attackers trained after it, and record the best result any of them achieves. On a scale from −1 (the defender always wins) to +1 (the attacker always wins), this best attacker result falls at every generation, from −0.28 to −0.70 (Table 3b). We will revise “keeps improving” to “improves consistently over the five generations.”

Q5. How much manual effort did the reward fix require? If the base model or attacker seed set changes, must new reward exploits be rediscovered and patched by hand?

The fixes took four rounds of inspection and patching, about 2 person-weeks in total; Appendix C lists the resulting rules. Once fixed, the reward needed no further patches: Our PSRO run on Qwen3-8B likewise used the same fixed reward throughout, and raised safety from .429 to .983 within four generations. Thus, changing the base model did not require rediscovering and patching exploits in this experiment, although other models or attacker seed sets may expose new ones. Reward hacking remains a general challenge when RL optimizes against a learned judge. Our contribution is population-based training; here, we use manual fixes to address the observed exploits and leave automated detection and reward refinement to future work.

Q6. Does the “comparable helpfulness” framing understate the helpfulness cost relative to the base model, and can the trade-off be stated explicitly?

We agree that “comparable helpfulness” understates the cost and will replace it with “a moderate helpfulness cost,” explicitly quantifying the trade-off: PSRO gains .599 in safety over the base model at a cost of .125 in helpfulness, concentrated on adversarially framed benign requests.

Q7. If a dominating defender is reachable, is the population providing coverage of a non-transitive strategy space, or mainly acting as an anti-forgetting mechanism? Does non-transitivity persist beyond five generations?

Thank you for this insightful question, which touches the core idea of the paper.

Non-transitivity and a dominating D5 are compatible. Non-transitivity describes the early iterations: newer defenders are not uniformly stronger. For example, attacker A1 does better against the newer D2 than against D1 (Table 3a: its payoff is −.53 vs. −.73, on a scale from −1, the defender always wins, to +1, the attacker always wins). Likewise, D3 is more vulnerable than D2 to A3, and D4 is more vulnerable than D2 and D3 to A2. No early defender handles every attacker, so the Nash equilibrium mixes several of them (Table 3b). The population keeps all of these attackers, and D5, trained against their mixture, resists all of them, so PSRO converges and the equilibrium concentrates on D5. Non-transitivity thus characterizes the early phase, and convergence the later one; the two do not contradict each other.

Coverage or anti-forgetting? In our view the population plays both roles: during training it supplies complementary attacks for the new defender to cover, and once a defender covers them, the retained attackers keep it from forgetting. Our current results do not allow us to conclude which of the two contributes more.

Does non-transitivity persist after five generations? We do not claim that it has disappeared. What we have found is a Nash equilibrium of the game restricted to the current population, which places all its weight on D5. Whether new non-transitive strategies would emerge over many more generations is beyond what our runs establish.

Thank you for the response. Most of my concerns and questions have been addressed, and I will raise my score accordingly.

Thank you for your time, constructive review, and for raising your score. We greatly appreciate your support of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.