- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes sequential population training for LLM reinforcement learning. An initial task-trained LoRA adapter is followed by additional LoRA adapters over a shared Qwen2.5-3B-Instruct backbone. Each new adapter receives task reward and a gated token-level cross-entropy score under frozen predecessor adapters. The authors evaluate four adapters on held-out BabyAI task families under prompt-history shifts and report higher population pass@k than independently trained PPO adapters in several shifted settings. A second experiment trains an abliterated Llama 3.1 8B jailbreak attacker and reports higher budget-matched coverage than GRPO and MaxRL on 1,000 tasks. The construction is interesting and the BabyAI results are potentially useful, but the manuscript does not establish semantic strategy change, decorrelated failure modes, or held-out jailbreak generalization. The formal objective and several displayed results also contain internal inconsistencies.
Critical strengths
1.Sections 3.3 to 3.4 define a dense score from frozen predecessor distributions and implement it with lightweight LoRA adapters. In the BabyAI study, the proposed population is compared with independently trained PPO adapters using one sample per policy and a nominally matched generation budget. This makes the observed population-coverage effect more informative than a comparison against a single model with a different sampling budget.
- The manuscript reports held-out task families and prompt-history shifts, includes the low-shift case where the method is not uniformly better, and provides diagnostics for surface-reward length inflation and passing-only versus all-trajectory diversity. These choices make the empirical question worthwhile even though they do not identify the proposed mechanism.
Critical weaknesses
-
Which directional objective was actually implemented? Please reconcile the argument order in Equations 1 to 3, state whether the intended objective is ordered or symmetric, and provide the exact loss used in code.
-
How many independent training runs underlie Figures 2 to 4 and the nonzero rows of Table 6? Please provide per-run results, paired uncertainty, checkpoint-selection rules, and the policy-order or subset rule used to compute population pass@k.
-
Are the 1,000 jailbreak tasks used for Figure 4 disjoint from attacker training and model selection? What exact reward, success judge, refusal definition, decoding settings, and training schedule produce the reported coverage?
-
What are the rolling-window length, first-adapter threshold, critic warmup,
D_targetmultiplier and value, smoothing status, and per-adapter measured task returns relative to each gate threshold?
Questions for authors
See in Weakness.
3 comments
[W2] Figure 2-4, Table 6 Training Details cont.
Jailbreaking results:
(1) Number of independent training runs. Fig. 4: one training run per method. Table 6: one run for each nonzero w_div; the w_div=0 row reports the mean and range over 5 runs.
(2) Per-run results. Fig. 4, per-member single-attempt success (held-out, %): Ours 66.2 / 83.0 / 75.9 / 82.7; GRPO 67.5 / 55.3 / 44.2 / 69.6; MaxRL 61.1. The four GRPO iterations are trained independently with different seeds. Our method's four iterations are trained in sequence with different seeds.
(3) Paired uncertainty. Fig. 4, Ours − GRPO with a 95% paired bootstrap over tasks: −1.2 [−2.8, 0.3] at B=1, +11.5 [10.4, 12.6] at B=2, +6.5 [5.8, 7.2] at B=4. This reflects evaluation uncertainty only, not retraining variance. Table 6: relative to the mean of the five w_div=0 runs, pass@2 differences are +1.1 [−0.5, 2.6] (w_div=0.05), +0.7 [−0.8, 2.2] (0.2), and −0.2 [−1.6, 1.2] (0.5) (95% paired bootstrap over tasks).
(4) Checkpoint selection. Fig. 4: final checkpoint (step 80) for all pool members, with no selection; MaxRL uses final checkpoint (step 320). Table 6: final checkpoint (step 40).
(5) Order / subset rule. Members are queried in the order listed in (2). For B ≤ 4, the first B members are queried once each; for B > 4, attempts are spread evenly over the 4 members (e.g., B=8: twice each). MaxRL spends all B attempts on itself.
[W3] Jailbreaking Task Specifications
Q: Are the 1,000 jailbreak tasks used for Figure 4 disjoint from the tasks used for attacker training and model selection?
Approximately one quarter of the original 1,000 evaluation tasks overlapped with training prompts, so the original evaluation was not fully held out. To address this, we re-evaluated all methods on a new held-out set of 1,000 prompts never used in training. The table reports coverage at budget B, defined as the probability that at least one of B attempts succeeds.
| B | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| MaxRL | 61.1 | 74.0 | 80.3 | 84.0 | 86.4 | 88.2 | 89.6 | 90.7 |
| GRPO | 67.5 | 80.9 | 85.7 | 92.7 | 95.8 | 96.7 | 97.2 | 98.1 |
| Ours | 66.2 | 92.4 | 97.4 | 99.2 | 99.5 | 99.8 | 99.9 | 100.0 |
At B=1, where each method uses a single policy, all three are comparable (61–68%). From B=2 onward, our pool outperforms both baselines. At B=2 it reaches 92.4%, compared with 80.9% for GRPO and 74.0% for MaxRL, which already exceeds MaxRL’s 90.7% at B=8. Our pool reaches 99.2% at B=4, a level GRPO does not reach even at B=8 (98.1%). As a check, we also evaluated all methods on 1,000 training prompts. Coverage differed from the held-out results by at most 1.3 percentage points for every method and budget, so we see no evidence of overfitting to training prompts.
Q: What exact reward, success judge, refusal definition, decoding settings, and training schedule produce the reported coverage?
Reward and refusal. We train the attacker to generate prompts that defeat the defender: the attacker is rewarded when its prompt induces the defender to produce a harmful response that fulfills the intended task. We follow the reward formulation of Self-RedTeam [1], using WildGuard [2] to label harmfulness and refusal, together with a format reward. We additionally use a HarmBench classifier [3] to veto harmfulness labels when the response does not match the intended behavior.
Success judge. An attack succeeds if WildGuard labels the defender’s response as harmful and the attack is well-formed. The HarmBench veto is used only for the training reward, not for evaluation.
Decoding. The attacker generates at temperature 1.0. The frozen defender (Mistral-7B-Instruct-v0.2) uses greedy decoding. Both have a 512-token limit.
Training. We fine-tune Llama-3.1-8B-Instruct-abliterated with LoRA. For GRPO and our method, each adapter is trained for 80 steps with the same hyperparameters and a different random seed. For MaxRL, we train a single model for 320 steps.
We will add the full configuration to the appendix.
References:
[1] Liu, Mickel, et al. "Chasing moving targets with online self-play reinforcement learning for safer language models." arXiv preprint arXiv:2506.07468 (2025).
[2] Han, Seungju, et al. "Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms." Advances in neural information processing systems 37 (2024): 8093-8131.
[3] Mazeika, Mantas, et al. "Harmbench: A standardized evaluation framework for automated red teaming and robust refusal." arXiv preprint arXiv:2402.04249 (2024).
[W4] Hyperparameter Specifications
(1) Rolling Window Length: We use a fixed 10-update rolling mean of the critic value to assess task performance, without additional EMA or post-hoc smoothing.
(2) First-Adapter Threshold: The first diversity adapter is the second adapter trained in the process, and has its performance threshold configured based on the critic value of the first baseline PPO adapter, yielding on average, around a 0.8 performance gate when training the first diversity adapter.
(3) The critic warmup procedure follows the default configuration with the initial larger rollout pool inherited from Verlog [4] and allows for the critic to converge to accurate estimates. The diversity gate remains closed for the first 20 updates of each diversity adapter's training phase to promote actor and critic stability. It subsequently activates when the rolling statistic exceeds the performance gate. The gate is reversible and deactivates diversity pressure if performance falls below this threshold.
(4) D_target: The diversity-reward cap, D_target, is set to be fixed at 1.6, with a sweep of diversity cap results presented in Appendix A.14, Table 8. Given our population sizes, we found a fixed target to be sufficient for encouraging reasonable amounts of diversity, but larger populations may need to have a dynamic target.
(5): Measured Task Returns: The mean of the observed rolling critic window following the 20-step warmup gate was 0.910, 0.883, and 0.931 for diversity adapters 1–3, respectively, with maximum observed values being 0.975, 0.943, and 0.966, respectively, as task performance still constitutes a large fraction of the objective, with the adapter task performance gates landing around 0.8 - 0.83 for successive adapters. Appendix A.14, Table 8 further reports the controlled sensitivity of gate activity and task performance to thresholds 0.70, 0.80, and 0.90.
References:
[4] Chen, Wentse, et al. "Verlog: Context-lite Multi-turn Reinforcement Learning framework for Long-Horizon LLM Agents." First Workshop on Multi-Turn Interactions in Large Language Models.
We thank the reviewer for identifying several important points requiring clarification and we address each concern below.
[W1] Which Directional Objective was Implemented
Our intended objective is directional rather than symmetric. Eq. 1 had an indexing typo: the summation should be over (1≤j<i≤K), rather than (1≤i<j≤K), while keeping D(πi,πj) unchanged. Trajectories are sampled from the current policy (πi), and their actions are evaluated under previously trained, frozen reference policies (πj), where (j<i). This convention is consistent with Equations 2–3 and the implementation, and has been corrected in the revised version of the paper.
For computing the loss, we use the standard clipped PPO objective with a dual clip bound for negative advantages. Advantages are calculated through a blend from two channels, where separate value heads estimate environmental and diversity advantages using GAE. These advantages are independently whitened over valid response tokens, combined, and whitened again: At=Whiten(wenvWhiten(Atenv)+wdivWhiten(Atdiv)) The resulting advantage enters the standard response-token-masked PPO actor objective, with clipping parameter ϵ=0.2 and dual-clip bound c = 3. Note we use wdiveff in place of wdiv when using the smoothing referred to in Appendix A.7.
Specifically, for each token sampled from the current policy, the implemented diversity reward is rtdiv=−i−11∑j<ilogπj(at∣st),at∼πi(⋅∣st). The reward is averaged across predecessor adapters, masked to response tokens, and clipped to ([0,5]) nats per token. This reward is then scaled by a constant factor and used exclusively for the diversity GAE calculations, and merged into the overall advantage according to the process described. More details are also provided in Appendix A.4.
[W2] Figure 2-4, Table 6 Training Details
BabyAI Results:
(1) Number of independent runs: The original Baby AI results from Figures 2-3, reflect one independent training run. We have since conducted additional independent BabyAI training runs with matched training seeds per iteration. We now report per-replication population coverage, mean ± standard deviation, and paired coverage differences across two replications (Appendix A.16, Figs. 8-9)
(2) Per-run results. We report per-replication population coverage in Appendix A.17, Table 10, with mean ± SD uncertainty plots in Appendix A.16, Figures 8–9. Under High OOD shift, the population coverage results for pass@k (%) are: for k = 1: PPO 28.90/26.60; CE-diversity 28.90/26.60, k = 2: PPO 47.65/42.95; CE-diversity 69.55/50.00, k = 3: PPO 58.60/57.85; CE-diversity 76.60/59.35, k = 4: PPO 75.00/66.40; CE-diversity 82.05/79.65. In each pair, the values correspond to independent training replications.
(3) Paired uncertainty. We compute CE-diversity minus PPO coverage within each matched training replication and report the mean ± sample SD of the two paired differences. Under High OOD shift: k=1: Paired differences 0.00/0.00 pp; mean ± SD = 0.00 ± 0.00 pp. k=2: Paired differences +21.90/+7.05 pp; mean ± SD = +14.48 ± 10.50 pp. k=3: Paired differences +18.00/+1.50 pp; mean ± SD = +9.75 ± 11.67 pp. k=4: Paired differences +7.05/+13.25 pp; mean ± SD = +10.15 ± 4.38 pp. Both replications favor CE-diversity for k>1 under medium and high OOD shifts, while low-OOD results are mixed. (Details reported in Appendix A.17, Table 10)
(4) Checkpoint selection: For evaluation, we use the final checkpoint at training step 150 for each policy, without selecting checkpoints based on test performance.
(5) Order/Subset Rule: Population pass@k is computed empirically rather than through a statistical pass@k estimator: we execute each of our k policies in sequential training order on the same evaluation instances, with one rollout per policy, and count an instance as successful if at least one policy succeeds. Thus, the reported coverage reflects observed complementary successes across policies under a fixed k attempt budget, rather than an extrapolation from sampled success probabilities.