acceptodds
Under review as a conference paper at ICLR 2027

Population-Based Diversity for Reinforcement Learning in LLMs via Sequential Multi-Adapter Optimization

10 views

Abstract

Reinforcement learning (RL) fine-tuning of large language models (LLMs) tends to collapse the policy onto a few strategies. This induces a severe quality-diversity (QD) trade-off: while single-policy performance improves, solution coverage (e.g., pass@) shrinks, leaving the model brittle when its preferred strategy fails. While other methods may opt to maintain diverse modes of behavior within a single policy, we instead use a framework that sequentially builds a population of LoRA adapters over a shared frozen backbone, optimizing each policy for task performance while rewarding distribution-level diversity against previously learned policies conditioned on sufficient task competence. Unlike response-level diversity objectives, our method compares the cross-policy probability distributions, avoiding surface-level similarity metrics or additional learned diversity models. On long-horizon BabyAI tasks, our method improves held-out pass@ and is more robust to prompt-based distribution shift than independently trained policies. We also demonstrate that our approach seamlessly transfers to large-scale jailbreak red-teaming

Primary area:
reinforcement learning
Keywords:
Quality Diversity, Reinforcement Learning, Population-Based Method

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_n15kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes sequential population training for LLM reinforcement learning. An initial task-trained LoRA adapter is followed by additional LoRA adapters over a shared Qwen2.5-3B-Instruct backbone. Each new adapter receives task reward and a gated token-level cross-entropy score under frozen predecessor adapters. The authors evaluate four adapters on held-out BabyAI task families under prompt-history shifts and report higher population pass@k than independently trained PPO adapters in several shifted settings. A second experiment trains an abliterated Llama 3.1 8B jailbreak attacker and reports higher budget-matched coverage than GRPO and MaxRL on 1,000 tasks. The construction is interesting and the BabyAI results are potentially useful, but the manuscript does not establish semantic strategy change, decorrelated failure modes, or held-out jailbreak generalization. The formal objective and several displayed results also contain internal inconsistencies.

Critical strengths

1.Sections 3.3 to 3.4 define a dense score from frozen predecessor distributions and implement it with lightweight LoRA adapters. In the BabyAI study, the proposed population is compared with independently trained PPO adapters using one sample per policy and a nominally matched generation budget. This makes the observed population-coverage effect more informative than a comparison against a single model with a different sampling budget.

  1. The manuscript reports held-out task families and prompt-history shifts, includes the low-shift case where the method is not uniformly better, and provides diagnostics for surface-reward length inflation and passing-only versus all-trajectory diversity. These choices make the empirical question worthwhile even though they do not identify the proposed mechanism.
Critical weaknesses
  1. Which directional objective was actually implemented? Please reconcile the argument order in Equations 1 to 3, state whether the intended objective is ordered or symmetric, and provide the exact loss used in code.

  2. How many independent training runs underlie Figures 2 to 4 and the nonzero rows of Table 6? Please provide per-run results, paired uncertainty, checkpoint-selection rules, and the policy-order or subset rule used to compute population pass@k.

  3. Are the 1,000 jailbreak tasks used for Figure 4 disjoint from attacker training and model selection? What exact reward, success judge, refusal definition, decoding settings, and training schedule produce the reported coverage?

  4. What are the rolling-window length, first-adapter threshold, critic warmup, D_target multiplier and value, smoothing status, and per-adapter measured task returns relative to each gate threshold?

Questions for authors

See in Weakness.

3 comments

We thank the reviewer for identifying several important points requiring clarification and we address each concern below.

[W1] Which Directional Objective was Implemented

Our intended objective is directional rather than symmetric. Eq. 1 had an indexing typo: the summation should be over (1≤j<i≤K1 \le j < i \le K), rather than (1≤i<j≤K1 \le i < j \le K), while keeping D(πi,πj)D(\pi_i,\pi_j) unchanged. Trajectories are sampled from the current policy (πi\pi_i), and their actions are evaluated under previously trained, frozen reference policies (πj\pi_j), where (j<ij<i). This convention is consistent with Equations 2–3 and the implementation, and has been corrected in the revised version of the paper.

For computing the loss, we use the standard clipped PPO objective with a dual clip bound for negative advantages. Advantages are calculated through a blend from two channels, where separate value heads estimate environmental and diversity advantages using GAE. These advantages are independently whitened over valid response tokens, combined, and whitened again: At=Whiten(wenvWhiten(Atenv)+wdivWhiten(Atdiv))A_t = \mathrm{Whiten}(w_{\mathrm{env}} \mathrm{Whiten}(A_t^{\mathrm{env}}) + w_{\mathrm{div}} \mathrm{Whiten}(A_t^{\mathrm{div}})) The resulting advantage enters the standard response-token-masked PPO actor objective, with clipping parameter ϵ=0.2\epsilon=0.2 and dual-clip bound c = 3. Note we use wdiveffw_{\mathrm{div}}^{\mathrm{eff}} in place of wdivw_{\mathrm{div}} when using the smoothing referred to in Appendix A.7.

Specifically, for each token sampled from the current policy, the implemented diversity reward is rtdiv=−1i−1∑j<ilog⁡πj(at∣st),at∼πi(⋅∣st).r_t^{\mathrm{div}} = -\frac{1}{i-1} \sum_{j<i} \log \pi_j(a_t \mid s_t), \qquad a_t \sim \pi_i(\cdot \mid s_t). The reward is averaged across predecessor adapters, masked to response tokens, and clipped to ([0,5]) nats per token. This reward is then scaled by a constant factor and used exclusively for the diversity GAE calculations, and merged into the overall advantage according to the process described. More details are also provided in Appendix A.4.

[W2] Figure 2-4, Table 6 Training Details

BabyAI Results:

(1) Number of independent runs: The original Baby AI results from Figures 2-3, reflect one independent training run. We have since conducted additional independent BabyAI training runs with matched training seeds per iteration. We now report per-replication population coverage, mean ± standard deviation, and paired coverage differences across two replications (Appendix A.16, Figs. 8-9)

(2) Per-run results. We report per-replication population coverage in Appendix A.17, Table 10, with mean ± SD uncertainty plots in Appendix A.16, Figures 8–9. Under High OOD shift, the population coverage results for pass@k (%) are: for k = 1: PPO 28.90/26.60; CE-diversity 28.90/26.60, k = 2: PPO 47.65/42.95; CE-diversity 69.55/50.00, k = 3: PPO 58.60/57.85; CE-diversity 76.60/59.35, k = 4: PPO 75.00/66.40; CE-diversity 82.05/79.65. In each pair, the values correspond to independent training replications.

(3) Paired uncertainty. We compute CE-diversity minus PPO coverage within each matched training replication and report the mean ± sample SD of the two paired differences. Under High OOD shift: k=1: Paired differences 0.00/0.00 pp; mean ± SD = 0.00 ± 0.00 pp. k=2: Paired differences +21.90/+7.05 pp; mean ± SD = +14.48 ± 10.50 pp. k=3: Paired differences +18.00/+1.50 pp; mean ± SD = +9.75 ± 11.67 pp. k=4: Paired differences +7.05/+13.25 pp; mean ± SD = +10.15 ± 4.38 pp. Both replications favor CE-diversity for k>1 under medium and high OOD shifts, while low-OOD results are mixed. (Details reported in Appendix A.17, Table 10)

(4) Checkpoint selection: For evaluation, we use the final checkpoint at training step 150 for each policy, without selecting checkpoints based on test performance.

(5) Order/Subset Rule: Population pass@k is computed empirically rather than through a statistical pass@k estimator: we execute each of our k policies in sequential training order on the same evaluation instances, with one rollout per policy, and count an instance as successful if at least one policy succeeds. Thus, the reported coverage reflects observed complementary successes across policies under a fixed k attempt budget, rather than an extrapolation from sampled success probabilities.

[W2] Figure 2-4, Table 6 Training Details cont.

Jailbreaking results:

(1) Number of independent training runs. Fig. 4: one training run per method. Table 6: one run for each nonzero w_div; the w_div=0 row reports the mean and range over 5 runs.

(2) Per-run results. Fig. 4, per-member single-attempt success (held-out, %): Ours 66.2 / 83.0 / 75.9 / 82.7; GRPO 67.5 / 55.3 / 44.2 / 69.6; MaxRL 61.1. The four GRPO iterations are trained independently with different seeds. Our method's four iterations are trained in sequence with different seeds.

(3) Paired uncertainty. Fig. 4, Ours − GRPO with a 95% paired bootstrap over tasks: −1.2 [−2.8, 0.3] at B=1, +11.5 [10.4, 12.6] at B=2, +6.5 [5.8, 7.2] at B=4. This reflects evaluation uncertainty only, not retraining variance. Table 6: relative to the mean of the five w_div=0 runs, pass@2 differences are +1.1 [−0.5, 2.6] (w_div=0.05), +0.7 [−0.8, 2.2] (0.2), and −0.2 [−1.6, 1.2] (0.5) (95% paired bootstrap over tasks).

(4) Checkpoint selection. Fig. 4: final checkpoint (step 80) for all pool members, with no selection; MaxRL uses final checkpoint (step 320). Table 6: final checkpoint (step 40).

(5) Order / subset rule. Members are queried in the order listed in (2). For B ≤ 4, the first B members are queried once each; for B > 4, attempts are spread evenly over the 4 members (e.g., B=8: twice each). MaxRL spends all B attempts on itself.

[W3] Jailbreaking Task Specifications

Q: Are the 1,000 jailbreak tasks used for Figure 4 disjoint from the tasks used for attacker training and model selection?

Approximately one quarter of the original 1,000 evaluation tasks overlapped with training prompts, so the original evaluation was not fully held out. To address this, we re-evaluated all methods on a new held-out set of 1,000 prompts never used in training. The table reports coverage at budget B, defined as the probability that at least one of B attempts succeeds.

B12345678
MaxRL61.174.080.384.086.488.289.690.7
GRPO67.580.985.792.795.896.797.298.1
Ours66.292.497.499.299.599.899.9100.0

At B=1, where each method uses a single policy, all three are comparable (61–68%). From B=2 onward, our pool outperforms both baselines. At B=2 it reaches 92.4%, compared with 80.9% for GRPO and 74.0% for MaxRL, which already exceeds MaxRL’s 90.7% at B=8. Our pool reaches 99.2% at B=4, a level GRPO does not reach even at B=8 (98.1%). As a check, we also evaluated all methods on 1,000 training prompts. Coverage differed from the held-out results by at most 1.3 percentage points for every method and budget, so we see no evidence of overfitting to training prompts.

Q: What exact reward, success judge, refusal definition, decoding settings, and training schedule produce the reported coverage?

Reward and refusal. We train the attacker to generate prompts that defeat the defender: the attacker is rewarded when its prompt induces the defender to produce a harmful response that fulfills the intended task. We follow the reward formulation of Self-RedTeam [1], using WildGuard [2] to label harmfulness and refusal, together with a format reward. We additionally use a HarmBench classifier [3] to veto harmfulness labels when the response does not match the intended behavior.

Success judge. An attack succeeds if WildGuard labels the defender’s response as harmful and the attack is well-formed. The HarmBench veto is used only for the training reward, not for evaluation.

Decoding. The attacker generates at temperature 1.0. The frozen defender (Mistral-7B-Instruct-v0.2) uses greedy decoding. Both have a 512-token limit.

Training. We fine-tune Llama-3.1-8B-Instruct-abliterated with LoRA. For GRPO and our method, each adapter is trained for 80 steps with the same hyperparameters and a different random seed. For MaxRL, we train a single model for 320 steps.

We will add the full configuration to the appendix.

References:

[1] Liu, Mickel, et al. "Chasing moving targets with online self-play reinforcement learning for safer language models." arXiv preprint arXiv:2506.07468 (2025).

[2] Han, Seungju, et al. "Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms." Advances in neural information processing systems 37 (2024): 8093-8131.

[3] Mazeika, Mantas, et al. "Harmbench: A standardized evaluation framework for automated red teaming and robust refusal." arXiv preprint arXiv:2402.04249 (2024).

[W4] Hyperparameter Specifications

(1) Rolling Window Length: We use a fixed 10-update rolling mean of the critic value to assess task performance, without additional EMA or post-hoc smoothing.

(2) First-Adapter Threshold: The first diversity adapter is the second adapter trained in the process, and has its performance threshold configured based on the critic value of the first baseline PPO adapter, yielding on average, around a 0.8 performance gate when training the first diversity adapter.

(3) The critic warmup procedure follows the default configuration with the initial larger rollout pool inherited from Verlog [4] and allows for the critic to converge to accurate estimates. The diversity gate remains closed for the first 20 updates of each diversity adapter's training phase to promote actor and critic stability. It subsequently activates when the rolling statistic exceeds the performance gate. The gate is reversible and deactivates diversity pressure if performance falls below this threshold.

(4) D_target: The diversity-reward cap, D_target, is set to be fixed at 1.6, with a sweep of diversity cap results presented in Appendix A.14, Table 8. Given our population sizes, we found a fixed target to be sufficient for encouraging reasonable amounts of diversity, but larger populations may need to have a dynamic target.

(5): Measured Task Returns: The mean of the observed rolling critic window following the 20-step warmup gate was 0.910, 0.883, and 0.931 for diversity adapters 1–3, respectively, with maximum observed values being 0.975, 0.943, and 0.966, respectively, as task performance still constitutes a large fraction of the objective, with the adapter task performance gates landing around 0.8 - 0.83 for successive adapters. Appendix A.14, Table 8 further reports the controlled sensitivity of gate activity and task performance to thresholds 0.70, 0.80, and 0.90.

References:

[4] Chen, Wentse, et al. "Verlog: Context-lite Multi-turn Reinforcement Learning framework for Long-Horizon LLM Agents." First Workshop on Multi-Turn Interactions in Large Language Models.

Reviewer_zrtVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a sequential population-based approach to quality-diversity RL for LLMs. It trains LoRA adapters to remain task-competent while differing from frozen predecessors at the policy-distribution level. Experiments on BabyAI and jailbreak red-teaming show improved population-level coverage, with great robustness to prompt-history shifts.

Critical strengths
  1. A useful way to model strategy diversity in RL. Representing different policies as LoRA adapters over a shared backbone provides a practical framework for learning a population of complementary strategies.
  2. Compatibility with different policy-gradient algorithms: The method works with both PPO and GRPO, making it useful across different RL training setups rather than tying its benefits to one optimizer.
Critical weaknesses
  1. Limited empirical validation: The evaluation uses a small subset of BabyAI tasks with one backbone and one setting for red-teaming. The main results also lack uncertainty estimates across independent training seeds.

  2. Limited evidence of strategy-level diversity: The paper shows improved coverage but does not directly demonstrate how the LoRA policies differ in their strategies. Since cross-entropy also depends on actor entropy, increased randomness or low-level output variation could potentially explain the gains.

Questions for authors
  1. Are the main improvements consistent across training seeds? Could you report uncertainty estimates?
  2. Do the LoRA policies learn different strategies? Could you show representative examples?
4 comments

We thank the reviewer for pointing out some limitations with empirical validation and strategy diversity which we address below.

[W1 / Q1] Are the main improvements consistent across training seeds? Are there uncertainty estimates?

We conducted an additional set of independent BabyAI training runs, with matched training seeds between both CE-Diversity and PPO adapter at every iteration, to evaluate the consistency and uncertainty estimates of our results. We now report mean population coverage and standard deviation across the two sets of runs under varying degrees of OOD shift (Appendix A.16, Figs. 8,9).

Fixed kMethodLow OODMedium OODHigh OOD
1PPO55.57 ± 1.3135.17 ± 7.7427.75 ± 1.63
1Ours55.57 ± 1.3135.17 ± 7.7427.75 ± 1.63
2PPO82.57 ± 3.6457.05 ± 5.5245.30 ± 3.32
2Ours83.67 ± 10.9274.22 ± 2.2359.77 ± 13.82
4PPO94.70 ± 3.1182.80 ± 1.0670.70 ± 6.08
4Ours94.55 ± 2.2687.90 ± 1.7080.85 ± 1.70

The results reflect consistent improvement in population coverage under medium and high OOD shifts, particularly at (k=2) and (k=4). Specifically, both independent replications show positive improvements of +14.85/+19.50 points under medium shift and +21.90/+7.05 points under high shift for k = 2 (Appendix A.17, Table 10). This suggests that the improvements are not driven by a single favorable training seed.

[W2] Strategy Level Diversity and Entropy Comparisons

Do coverage gains reflect strategy-level diversity rather than increased stochasticity?

To examine strategy-level differences, we introduce a trajectory-level analysis in Section 4.7 (Figure 5). We first use GPT-6 to identify recurring strategy types from representative successful and failed trajectories, then use GLM-5.3-Flash to classify trajectories across seeded BabyAI evaluations. We analyze new instances of successes by the current adapter but not its predecessors to measure complementary task-solving behavior.

In particular, diversity-trained populations accumulate more successful non-loop trajectories than independent PPO (254 vs. 224, 294 vs. 276, and 323 vs. 304 across successive policy positions; Figure 5a). For diversity adapters, the newly contributed successes also span a more even distribution of different behavioral categories with respect to that of PPO, including navigation, search, and recovery strategies (Figure 5b). These results provide descriptive evidence that the coverage gains are accompanied by structured behavioral differences, and the issue of stochasticity is addressed following.

Distribution of Behavioral Strategy Categories Among Newly Contributed Successful Trajectories Across Successive Adapter Additions

AdditionMethodnDirectSearchScanWall RouteRoamingLoop-heavy
1Ours3946.2%33.3%0.0%17.9%0.0%2.6%
1PPO3823.7%36.8%0.0%31.6%0.0%7.9%
2Ours4619.6%21.7%6.5%13.0%0.0%39.1%
2PPO2055.0%10.0%0.0%25.0%5.0%5.0%
3Ours2343.5%13.0%0.0%39.1%0.0%4.3%
3PPO2138.1%0.0%0.0%57.1%4.8%0.0%

Normalized Behavioral Strategy Entropy Across Successive Adapter Additions

AdditionPer-addition: OursPer-addition: PPOΔCumulative: OursCumulative: PPO
10.6280.711-0.0830.6280.711
20.8160.673+0.1430.8140.746
30.6310.465+0.1670.8070.727

Although PPO has higher entropy at the first addition, our method produces substantially more diverse strategy distributions at the second and third additions. Consequently, cumulative entropy (reflecting pooled category counts) overtakes PPO after the second adapter and remains higher after the third addition, indicating the broadening of the population’s behavioral repertoire as additional adapters are introduced.

With regard to actor entropy, we further compare token-level actor entropy between CE-diversity and seed-matched standard PPO policies (Appendix A.15 Table 9). Both methods undergo substantial entropy reduction during training, from approximately 0.8 initially to mean final values of 0.0868 (CE-diversity) and 0.0486 (PPO), without any additional explicit entropy regularization. Thus, both methods learn substantially sharpened token-level action distributions, with a relatively small absolute difference in final entropy (0.0383, approximately 4.8% of the initial entropy). Although CE-diversity retains higher final entropy, entropy alone does not characterize strategy-level differences; our trajectory analysis provides complementary evidence of differences in task-solving behavior.

Q2: Concrete Examples of Strategic Gains and Breakaway from Failure Modes

Specific examples of wider strategic coverage that help avoid looping and other failure modes can be found in Appendix A.18 Figure 10. We present three matched BabyAI instances where independent PPO adapters fail whereas a diversity-trained adapter succeeds. In an OPEN task, PPO repeatedly moves forward after acquiring the key, whereas the diversity adapter follows an alternative wall route and successfully opens the door. In a PICKUP->GOTO task, PPO prematurely targets the destination and enters a failed-pickup loop, whereas the diversity adapter acquires the prerequisite object, verifies that it is carrying it, and then retargets the destination. In another OPEN task, PPO attempts the door before obtaining the key and becomes stuck in an interaction loop, while the diversity adapter retrieves the key, returns to the door, and completes the task.

These examples illustrate differences in prerequisite sequencing, navigation, and recovery from repetitive failure modes. Together with the broader composition of newly contributed successful behaviors in Figure 5(b), they suggest that successive diversity-trained adapters learn complementary task-solving behaviors rather than merely producing different low-level action sequences.

Thanks for the response and the additional experiments and examples. These address mosdt of my concerns. I will adjust my score accordingly.

Reviewer_FrybOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a population-based approach to diversity in LLM reinforcement learning. It sequentially trains LoRA adapters over a shared frozen backbone, rewarding each new adapter for producing behavior unlikely under previously trained adapters. A performance gate controls the diversity incentive to preserve task competence. Experiments on BabyAI and jailbreak red-teaming demonstrate improved population-level coverage and robustness to distribution shift.

Critical strengths
  1. The method provides a interesting way to encourage complementary policies without predefined behavior categories or an additional learned diversity discriminator.
  2. Comparisons against independently trained adapter populations help isolate the contribution of explicit diversity training, with promising coverage improvements under distribution shift.
Critical weaknesses
  1. My main concern is computational cost. LoRA reduces parameter storage, but this does not necessarily imply inexpensive training. Training multiple adapters and evaluating their outputs under previously trained adapters introduce additional computation. Multi-adapter rollout generation also adds serving and scheduling complexity. Matching the number of generated attempts is therefore insufficient to establish overall efficiency. The paper would benefit from comparisons of training GPU-hours, reference-scoring overhead, peak memory, and inference latency or throughput, together with practical insights into reducing these costs.

  2. Scalability to larger populations remains unclear. When every new adapter is compared with all predecessors, the total number of reference-policy comparisons grows quadratically with population size under a fixed per-adapter training budget. Inference requires additional adapter management, although the generation budget itself need not grow superlinearly. By contrast, diversity maintained within a single policy can be exploited by increasing the number of sampled rollouts without training additional policies. Experiments showing coverage versus total compute as the population grows would clarify where the proposed approach remains competitive.

  3. The evidence primarily establishes population-level gains rather than improvements in individual-policy capability. The approach is particularly suitable for settings that permit multiple attempts and provide a reliable success criterion. Its benefits are less clear when only one policy can be deployed or one answer can be returned, such as pass@1 evaluation on mathematical reasoning tasks. Evaluating individual adapters, or explaining how to select an adapter without access to outcome feedback, would better establish the method’s applicability beyond multi-attempt coverage.

Questions for authors

Minor:

  1. The red-teaming section states that all adapters train on the same 1,000 tasks and reports evaluation on 1,000 tasks. Are these the same tasks or disjoint splits? Please clarify whether the reported coverage measures training-task coverage or held-out generalization.

  2. How sensitive is the method to the performance-gate threshold and diversity cap? Since the gate relies on critic estimates, how accurately does it track actual task success across training phases?

6 comments

We thank the reviewer for highlighting the computational trade-offs of population-based diversity training. We agree that, although LoRA reduces parameter storage, scoring trajectories against all previously learned adapters introduces additional training computation.

[W1] Referential Scoring Overhead Concerns

We have conducted further analysis to address all of the concerns regarding computational overhead due to the referential scoring. Across our four-policy BabyAI population, total training requires 276.68 NVIDIA A6000 GPU-hours. Of this, approximately 13.52 GPU-hours (4.88%) are attributable to predecessor-policy scoring. As the number of predecessors increases from one to three, reference scoring requires 3.81%, 7.32%, and 10.45% of step time, respectively. The cost is higher at 8B scale in jailbreak red-teaming, because each frozen predecessor needs a full forward pass over long attacker sequences on the same GPUs used for training. On 2×H200, a GRPO adapter takes 70.2 s per step, while our adapters take 92.7, 95.3, and 100.9 s with 1, 2, and 3 references (+32%, +36%, +44%). Training the four-adapter pool takes 9.0 h versus 6.4 h for the GRPO pool (+41%; 18.5 vs. 13.2 GPU-hours). In both domains the cost grows roughly linearly with the number of references.

Reference scoring adds little memory, because the frozen predecessors are LoRA adapters. In BabyAI, peak allocated memory matches the PPO baseline (30.05 GB vs. at most 30.28 GB). In jailbreak red-teaming, the actor’s peak allocated memory is identical to GRPO (16.9 GB), and each additional reference adapter adds only about 0.6 GB to the reference model. At inference time, our method serves a single trained adapter and does not evaluate any predecessor policies; therefore, it introduces no referential scoring overhead or additional inference-time computation and forward passes relative to the PPO or GRPO baseline, while overall latency may vary slightly depending on adapter routing and serving implementations. Further detailed results for Baby AI are reported in Appendix A.12, Table 7.

In jailbreak red-teaming, step time grows by a fixed ~20 s plus roughly 4 s per additional reference (on a 70 s GRPO step). Extrapolating linearly, a population of 10 would cost about 50% more than independent GRPO training in total, with the last adapter about 75% slower per step. Our approach therefore scales comfortably to small and moderate populations; we agree that scaling to populations on the order of 100 would require bounding the reference set (see W2).

Specifically, in BabyAI, averaging across all tasks, our two-policy population achieves 85.94% coverage with 126.55 GPU-hours, whereas independently trained PPO needs three policies and 162 GPU-hours to reach comparable coverage (>85%).

[W2] Coverage Compute Tradeoff, Scalability, and Comparison with Single Policy Diversity Methods

Coverage vs Total Compute:

We present additional coverage vs training compute graphs for both Baby AI and Jailbreak Red Teaming tasks in Appendix A.19 Figure 11. On BabyAI, our method achieves approximately 73% versus 62% coverage at k = 2 with similar compute, and 88% versus 83% at k = 4, albeit with higher training cost. For red-teaming, our three-policy population achieves approximately 97% coverage at 13 H200 GPU-hours, compared with 93% for four GRPO policies at comparable compute. These results demonstrate favorable coverage–compute trade-offs for small populations, while acknowledging increasing predecessor-scoring overhead as population size grows.

Comparison with Single-Policy QD Methods:

With regard to single policy QD, we would like to highlight that our method is complementary with single-policy QD-RL methods like MaxRL which optimizes for pass@k across multiple attempts from the same policy. In principle, our cross entropy diversity objective could be incorporated into these approaches, encouraging each additional policy to complement previously trained policies. This combination could potentially achieve higher coverage with fewer policies, although we have not yet tested this empirically.

Scalability:

The quadratic cost arises only if every new adapter is scored against all previous ones. In practice, the reference set only needs to cover the distinct strategies the population has already found. If a task admits roughly m distinct strategies (the width of the strategy space in the sense of Czarnecki et al. [1]), adapters beyond the m-th mostly revisit existing modes, so a rolling window of the m most recent adapters is sufficient and total cost grows as O(Km)O(Km) rather than O(K2)O(K^2). Our jailbreak results suggest m is small: four adapters already reach 99.2% held-out coverage at k=4. In general m is unknown in advance; it can be chosen by growing the window until held-out coverage stops improving. We have not evaluated this at large K and will list it as future work.

References:

[1] Czarnecki, Wojciech M., et al. "Real world games look like spinning tops." Advances in Neural Information Processing Systems 33 (2020): 17443-17454.

[W3] Concerns over effectiveness in single attempt evaluation settings

Our method is primarily designed for settings where multiple attempts or complementary policies are useful, aiming to improve population-level pass@k coverage, rather than producing a single policy optimized for pass@1. This formulation is consistent with the motivation of learning a diverse set of policies in order to adequately respond to OOD settings, with potential extensions adjacent to SMERL [2] for selecting appropriate policies, although such a mechanism is outside the scope and claims of our work.

Q1: Are the 1,000 red-teaming evaluation tasks the same as the training tasks, or a disjoint split? Does the reported coverage measure training-task coverage or held-out generalization?

Approximately one quarter of the original 1,000 evaluation tasks overlapped with training prompts, so the original evaluation was not fully held out. To address this, we re-evaluated all methods on a new held-out set of 1,000 prompts never used in training. The table reports coverage at budget B, defined as the probability that at least one of B attempts succeeds.

B12345678
MaxRL61.174.080.384.086.488.289.690.7
GRPO67.580.985.792.795.896.797.298.1
Ours66.292.497.499.299.599.899.9100.0

At B=1, where each method uses a single policy, all three are comparable (61–68%). From B=2 onward, our pool outperforms both baselines. At B=2 it reaches 92.4%, compared with 80.9% for GRPO and 74.0% for MaxRL, which already exceeds MaxRL’s 90.7% at B=8. Our pool reaches 99.2% at B=4, a level GRPO does not reach even at B=8 (98.1%). As a check, we also evaluated all methods on 1,000 training prompts. Coverage differed from the held-out results by at most 1.3 percentage points for every method and budget, so we see no evidence of overfitting to training prompts.

Q2: Sensitivity of performance gate and diversity cap / Critic accuracy

We conducted additional sensitivity sweeps (Appendix A.14, Table 8). The results highlight the roles of both hyperparameters in balancing task performance and behavioral diversity. For the diversity cap, a more restrictive setting (0.8) suppresses the diversity incentive, while a more permissive setting (3.2) encourages greater behavioral divergence, with some reduction in task performance. Similarly, a lower performance-gate threshold (0.70) introduces diversity incentives earlier, whereas a higher threshold (0.90) delays their activation. Our default configuration (gate ~ 0.80, cap = 1.6) achieves the best observed overall pass@1 (91.02%), supporting the importance of appropriately balancing task competence and diversity, and can be reasonably estimated off of metrics from a baseline run.

Regarding critic estimates, all runs involve a critic warm-up stage with gate activation disabled for the first 20 updates, to promote task performance and actor / critic stability before introducing diversity terms. The mean post-warm-up critic loss is 0.00180, decreasing to 0.00083 during late training. Comparing critic estimates against actual in-distribution validation success yields a mean absolute error of 0.0965, with estimation error decreasing across training phases. These results suggest that the critic provides an increasingly accurate estimate of task performance.

References:

[2] Kumar, Saurabh, et al. "One solution is not all you need: Few-shot extrapolation via structured maxent rl." Advances in Neural Information Processing Systems 33 (2020): 8198-8210.

Comments on the paper

1 comment

Dear Reviewers,

Thank you very much for your constructive feedback and thorough review of our manuscript.

We hope our previous response and revisions have satisfactorily addressed all of your concerns. As the discussion period is coming to an end soon, we would greatly appreciate it if you could kindly let us know whether our response has resolved your concerns, or if there are any remaining points we could clarify further before the deadline.

Thank you for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.