acceptodds
Under review as a conference paper at ICLR 2027

Population-Based Post-Training for Large Language Models

Abstract

Reinforcement learning with verifiable rewards (RLVR) is one of the most popular post-training paradigms for large language models. RLVR typically trains a single model under hyperparameters chosen before training begins, while researchers tune hyperparameters across consecutive runs. We introduce **P**opulation-**B**ased **P**ost-**T**raining (PBPT), which trains several models in parallel and lets the training process do the tuning through population-based training. PBPT periodically evaluates models on a validation set. High-performers are duplicated and their hyperparameters perturbed to explore new configurations. Low-performers stop training and are replaced by these copies. On reasoning benchmarks (Enigmata, Nemotron) and interactive tool-use environments (Gaia2, AppWorld), PBPT produces stronger agents than compute-matched groups of independent RLVR learners. Furthermore, PBPT is able to recover from poor initial hyperparameters, automatically discovering better settings over the course of training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
population-based training, large language models, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6YSGOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

An evolutionary algorithm for hyper parameter optimization during post training.

Critical strengths

The use of mutation and crossover and the weight sharing during post-training are creative and interesting.

Critical weaknesses

None

Questions for authors

Does the mutation ablation include crossover or not? Generally, I would have liked to see more on the efficacy of the crossover mechanism. I'd also be interested to learn more about how you set the various thresholds and whether they are even needed. For example, you could choose always to copy weights in all tournaments, and always to perturb anyway. Same goes for the evaluation schedule spacing. Did you test starting from human presets (with some perturbations) versus an all random starting point?

3 comments

We sincerely thank you for your support and positive feedback. We are glad that you find our use of mutation, crossover, and weight sharing during post-training "creative and interesting." We address your questions below.

Crossover and the Mutation Ablation

We appreciate your interest in the crossover mechanism.

In the mutation ablation, we disable initial and subsequent mutation, so all learners retain the same hyperparameter configuration. Crossover remains enabled, but combining identical configurations has no effect.

In full PBPT, crossover plays a key role. Configuration diversity is important for exploring the hyperparameter search space effectively. Simply copying the donor's entire configuration at each replacement would reduce this diversity. Crossover combines donor and recipient hyperparameters, which allows successful settings to spread while retaining variation across learners.

Decision and Evaluation

Weight replacement threshold (why not always copy weights?) In our early experiments, we found that checkpoints with a temporary fitness advantage did not necessarily lead to the best final policies. For example, mutation towards a more aggressive configuration, such as a higher learning rate, could yield rapid early gains but lower final performance. Requiring a sufficient fitness gap before copying weights reduces the risk of such checkpoints dominating the population prematurely.

Mutation threshold (why not always perturb anyway?) Always perturbing would make it difficult to retain a good configuration long enough to benefit from it and propagate it through selection. The mutation threshold allows a learner to continue training with its current configuration when its fitness gap is small.

Winner and loser proportions. We divide the population approximately equally into winners, keepers, and losers. For our 8-member population, we use a 3/3/2 split, allocating the extra member to the winner set to increase donor diversity while keeping the loser set smaller to reduce unnecessary weight replacements. These proportions were chosen heuristically and can be considered hyperparameters of the method.

Warm-up and cooldown. Warm-up and cooldown are motivated by our prior experience with PBT. A newly mutated configuration needs time to influence training before its effectiveness can be assessed. Warm-up provides this time at the start of training, while cooldown does so after a configuration change.

Evaluation schedule. Evaluating too frequently consumes compute that would otherwise be available for training. Evaluating too infrequently provides less feedback for population decisions and too few data points in plots such as Figure 2. We choose the evaluation intervals to balance these considerations.

Initial Configurations

We evaluated a suboptimal initial configuration on Enigmata in addition to the human-tuned configurations used in our main experiments. See the PBPT and RLVR results from Table 1 of the revised manuscript below, averaged over the final 20% of the training budget. The following table reports mean test performance (%) across learners and for the learners with the highest and lowest validation means, respectively.

Initial configurationMethodAvg. testBest-val. testWorst-val. test
TunedRLVR75.4078.0073.03
TunedPBPT77.5978.1277.03
SuboptimalRLVR40.8745.8626.70
SuboptimalPBPT76.4076.7475.79

PBPT's advantage over independent RLVR is much larger under the suboptimal initialization. PBPT approaches its performance from the tuned initialization, with a small remaining gap, whereas independent RLVR degrades severely. The suboptimal configuration was still chosen by a human engineer to be intuitively reasonable, but without iterative tuning. We have not evaluated PBPT with its entire population initialized randomly from the search space. Our search space is deliberately broad, so such sampling can produce configurations that make little progress within the training budget, such as learning rates below 10−710^{-7}. We would expect mutations to give PBPT an even larger advantage over fixed-configuration RLVR in that setting, but we consider the comparison against plausible initial configurations more informative.

We hope our responses address your questions, and we thank you again for your support of our work.

Thank you for the explanations and revisions. The paper looks much stronger for it.

Thank you for your recognition and support for our work!

Reviewer_yGgLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts population-based training, a well-known model optimization algorithm for training neural networks for post-training language models. The authors demonstrate that executing population-based training while recording RLVR parameters and hyperparameters as states helps modestly improve RLVR performance for Qwen3.5-4B on 3 benchmarks and Qwen3.5-9B for 1 benchmark.

Critical strengths

This paper introduces population-based model optimization for RLVR. Several recent RLVR algorithm papers do not perform extensive RL hyperparameter optimization, which makes the paper’s idea of attempting population-based training a valuable optimization setting for researchers testing such algorithms. This work suggests that RLVR researchers should ensure gaps in policy performance are not simply due to poor RLVR hyperparameter selection by demonstrating significant performance improvements over suboptimal initial configurations simply by incorporating population-based training.

Critical weaknesses
  1. The current approach described in the paper has incremental novelty as it does not modify the overall population-based post-training at all beyond implementing the RLVR hyperparameters and policy parameters as a state. I do not feel that the paper in its current form meets the novelty bar for an ICLR paper - it only claims that population-based training can help "recover” RLVR performance from a suboptimal starting configuration without specifying how the suboptimal configuration was selected or showing that such configurations are widespread in RLVR training. The paper thus reads as an application of a well-known optimization algorithm to RL hyperparameter optimization.
  2. The experimental results are only compared to RLVR runs intialized from "suboptimal” initial configurations. Currently, the paper does not provide comparisons to any other hyperparameter optimization techniques. Consequently, despite reporting performance gains of ~5-15%, these gains are sensitive to how these suboptimal configurations were selected. Additionally, these comparisons were only made on RLVR experiments on Qwen3.5-4B (and Qwen3.5-9B for one task), which makes it unclear if these results generalize to different models or model families.

*The 5-15% estimation is based on my reading of Figure 2, I did not find an experimental results table with the final performance numbers for non-Enigmata experiments.

Questions for authors
  1. In Table 2, there are several RL hyperparameters in the initial configuration, some of which are fixed for Nemotron and some that are fixed for Enigmata. Why were hyperparameters fixed for some tasks and not others?
  2. Can the authors describe the impact on performance if a greater proportion of hyperparameters were fixed? How much of the reported performance increases are driven by optimizing hyperparameters with lower projection coefficients in Table 6? It seems that simplifying the state would speed up training and may not sacrifice as much performance if these hyperparameters are not significantly impacting the results.
  3. Why are no other hyperparameter optimization methods compared against? Do authors expect results to change drastically if a standard evolutionary search algorithm was applied only to the RL hyperparameter search space (instead of PBT)?
  4. How were the suboptimal initial configurations selected?
12 comments

Thank you for your review and constructive feedback. We sincerely appreciate your recognition of our population-based training as "a valuable optimization setting" for researchers studying RLVR algorithms. Below, we address your concerns and questions point by point.

W1 and W2: Baseline Configurations and Novelty

We would like to first clarify that all four main comparisons in Section 5.1 and Figure 2 of the revised manuscript start from human-tuned RL configurations, instead of suboptimal configurations. We manually adjusted the hyperparameters over successive trial training runs on each benchmark. The "performance gains of 5-15%" you observed in Figure 2 are actually improvements over independent RLVR with already human-tuned initial configurations. The suboptimal-configuration experiment in Section 5.2 is an additional study of recovery on Enigmata, not the setting for the majority of our results.

Regarding novelty, we agree that PBPT applies the core ideas of PBT to LLM RLVR. However, we note that our work provides one of the first systematic studies of PBT in RLVR that demonstrates its effectiveness on multiple benchmarks, despite the wide adoption of RLVR in LLMs. Making PBT effective in this setting requires addressing unique challenges. To better present these contributions, we have fully rewritten Sections 3 and 4 to explain what challenges are and the resulting algorithmic differences from prior work about PBT. Specifically, in PBPT, we compare checkpoints at matched compute to account for configuration-dependent training costs. We require a larger fitness gap for weight replacement than for mutation, allowing configuration changes without prematurely discarding useful policies under noisy validation. We also allow mutation operators to jointly explore continuous and discrete RLVR settings, including the RL objective.

We also agree that broader model coverage is important for assessing PBPT's effectiveness. To address this concern, we include an additional Nemotron comparison with Qwen3.5-9B in the revised manuscript, using the same setup as the 4B comparison. We follow the protocol in Table 1 of the revised manuscript, averaging evaluations over the final 20% of the 66-hour budget. The following table reports mean test performance (%) across learners and for the learners with the highest and lowest validation means, respectively. Appendix B.2 of the revised manuscript provides the full results and experimental details.

MethodAvg. testBest-val. testWorst-val. test
RLVR54.9959.4448.61
PBPT63.8364.1164.04

For comparisons with other hyperparameter optimization methods, please see our response to Q3 below.

Q1: Fixed Hyperparameters For Replay and Concurrency

Thanks for the careful reading. The differences are mainly in scenario replay and concurrency parameters.

Scenario replay is enabled by default in our implementation. It prioritizes previously attempted tasks with success rates that are not close to 0 or 1. Nemotron, along with AppWorld and Gaia2, retains this default. Enigmata, however, has a generator that can produce arbitrarily many training tasks. We therefore disable replay for Enigmata and instead enable per-family sampling weights, adapting the training distribution over task families rather than individual tasks.

Concurrency parameters control the number of parallel scenario runners during training and evaluation. They do not change the RL update rule, but can affect system throughput. Poor choices can reduce how much training fits within a fixed compute budget, so we keep these parameters at manually chosen values by default.

Enigmata has the simplest execution harness among our benchmarks, consisting of single-turn question answering. This makes it straightforward to vary the number of parallel runners without significant resource contention. In contrast, other benchmarks involve complex tool use, virtualized environments, or LLM-based judging, where increasing concurrency can introduce substantial memory and resource overhead. We therefore allow PBPT to optimize concurrency on Enigmata but keep it fixed at manually selected values elsewhere. This is primarily a systems-level consideration that modestly affects training throughput rather than the underlying RL algorithm.

Q2: Fixed Hyperparameters and Visualization Coefficients

We would like to clarify that the coefficients in Table 5 of the revised manuscript (Table 6 in the original submission) are used only to construct the axes for qualitatively visualizing PBPT's training trajectories in Figures 5(c) and 8. These coefficients play no role at all in training, selection, or mutation and therefore cannot account for any of the reported performance gains. In particular, a small projection coefficient does not imply that a hyperparameter has little effect on training.

Regarding fixing a greater proportion of hyperparameters, selecting which hyperparameters to mutate and defining their search ranges involves an important trade-off. At one extreme, fixing all hyperparameters eliminates configuration exploration, reducing PBPT to selection and state transfer. At the other extreme, overly broad search spaces can waste substantial compute on unpromising configurations. Since it is difficult to predict which hyperparameters will yield the greatest improvements, we adopt a middle ground: allowing sufficient flexibility to discover better configurations while restricting search ranges to avoid known unproductive regions.

To explore how hyperparameter settings relate to performance, we divided each PBPT run into 6-hour training windows. Within each window, we averaged each learner's hyperparameter values and validation scores and computed their Spearman rank correlation across the 8 learners. We then averaged these correlations across windows, so a positive correlation indicates that learners with higher parameter values tend to score higher at comparable training times. We find out that the associations vary across benchmarks. For example, evaluation temperature is positively correlated with validation performance on Nemotron and Enigmata, with average correlations of approximately 0.4 to 0.5, but is only weakly correlated on AppWorld. This variation supports our choice to keep multiple hyperparameters mutable, allowing PBPT to adapt training configurations to each task.

Q3: Comparisons with Other Hyperparameter Search Methods

We agree that comparisons with other hyperparameter optimization methods would strengthen the evaluation. Following your suggestion, we have run a random-search baseline on Enigmata, Nemotron, and AppWorld. Our implementation of random search uses PBPT's initial mutation to sample configurations around the human-tuned starting point and then trains each learner independently. For these comparisons, we train Qwen3.5-4B with the same number of learners and per-learner training budget as PBPT.

We follow the protocol in Table 1 of the revised manuscript, averaging evaluations over the final 20% of the 66-hour budget. The following table reports mean test performance (%) across learners and for the learners with the highest and lowest validation means, respectively.

BenchmarkMethodAvg. testBest-val. testWorst-val. test
EnigmataRandom search75.8277.4370.86
EnigmataPBPT77.5978.1277.03
NemotronRandom search45.8960.0136.69
NemotronPBPT60.5762.0360.85
AppWorld (normal)Random search75.6172.7766.67
AppWorld (normal)PBPT84.7386.9080.44
AppWorld (challenge)Random search52.6951.1447.91
AppWorld (challenge)PBPT63.1864.2467.15

PBPT outperforms random search across all three benchmarks and all three reported measures.

We also ran Successive Halving (SH) on Nemotron with Qwen3.5-4B. It starts with 16 learners: 8 use PBPT's initial configurations, and 8 use configurations sampled from PBPT's full search space. The three rounds train 16, 8, and 4 learners for 20 hours each. After each of the first two rounds, we retain the half with the highest mean validation scores over the last 4 hours. Surviving learners continue from their own checkpoints with unchanged configurations. Each learner uses 4 B200 GPUs, and the total GPU budgets are comparable: 2,240 GPU-hours for SH and 2,112 for PBPT. We use the same final-20% averaging protocol, over hours 48 to 60 for the 4 SH survivors and 52.8 to 66 for the 8 PBPT learners.

MethodAvg. testBest-val. testWorst-val. test
SH55.3958.5051.54
PBPT60.5762.0360.85

PBPT also outperforms SH across all three measures. These results support PBPT's advantage over both random search and SH under comparable compute budgets.

Q4: Choice of Suboptimal Initial Configurations

In this supplementary Enigmata experiment from suboptimal initial configurations, we asked a human engineer to choose an intuitively reasonable configuration without tuning it through trial runs. Compared with the tuned configuration, it uses a higher learning rate, more permissive clipping bounds, and more optimization epochs per iteration, leading to more aggressive updates. The configuration is within PBPT's search ranges. This experiment aims to test PBPT's ability to recover when an intuitively chosen configuration turns out to be suboptimal. We note that the four main comparisons in Section 5.1 and Figure 2 use human-tuned initial configurations instead.

We hope our responses address most of your concerns, and we would appreciate your reconsideration of our work in light of the new results and manuscript revisions.

Reviewer_3UxBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Population-Based Post-Training (PBPT), which applies population-based training to reinforcement learning with verifiable rewards (RLVR) for large language models. PBPT maintains multiple asynchronously trained learners and uses validation performance as fitness. Through selection, training-state transfer, hyperparameter crossover, and mutation, weaker members inherit progress from stronger members while exploring training configurations online. The authors compare PBPT against compute-matched independent RLVR runs on Enigmata, Nemotron, Gaia2, and AppWorld. They also investigate recovery from suboptimal initialization, the role of mutation, and the evolution of training configurations.

The paper addresses an important problem, presents a well-developed system, and demonstrates useful improvements in performance and training stability. The main limitations are that the current comparisons do not sufficiently disentangle the contributions of initial configuration diversity, online search, and training-state transfer, and that the reliability of validation fitness and statistical uncertainty across independent population runs remain insufficiently analyzed.

Critical strengths
  1. RLVR is sensitive to training configurations, and individual runs can experience substantial degradation. Independent experiments typically cannot share the progress they have already made. PBPT offers a practical approach to improving the reliability of parallel post-training and reducing resources wasted on unsuccessful runs.
  2. The paper describes compute-matched peer checkpoint comparisons, decentralized decisions, historical donor selection, model and optimizer state transfer, and cooldown periods after configuration changes. These mechanisms accommodate differences in training speed, staggered starts, and temporarily unavailable members. The main text and appendix provide substantial implementation detail.
  3. The four benchmarks span logical reasoning, mathematics and programming, instruction following, and multi-turn tool use. The main comparisons match the base model, number of learners, and per-learner training resources, while keeping the test sets separate from population decisions. The results suggest that PBPT’s population-level gains are not confined to a single task.
  4. Under suboptimal initialization on Enigmata, full PBPT, selection-only PBPT, and independent RLVR achieve average test accuracies of 76.40%, 70.07%, and 40.87%, respectively. These results support two observations: transferring successful training states can mitigate degradation, and configuration mutation can further help training escape poor configurations. The comparable performance of the two PBPT variants under tuned initialization also provides useful evidence about when mutation is beneficial.
  5. In addition to population averages, the paper reports the test performance of members ranked highest and lowest by validation performance. This helps distinguish improvements in the strongest model from improvements in training reliability. The hyperparameter trajectories illustrate changes in learning rates, rollout counts, and RL objectives, providing useful observations for further study of adaptive training configurations.
Critical weaknesses
  1. The lack of a baseline with matched initial configuration diversity limits attribution of the gains.
    Full PBPT changes the configurations of some members at initialization, whereas independent RLVR runs share the same initial configuration. The comparison therefore combines the effects of initial configuration diversity, online mutation, and state sharing. This is particularly consequential in the suboptimal-initialization experiment, where fixing all independent runs to the same unstable configuration may not represent a reasonable tuning strategy under the same budget. The most direct additional baseline would initialize independent RLVR runs with the same configuration set used by PBPT and then train them independently. The existing no-mutation ablation disables both initial and online mutation, so these effects also need to be separated. Random search or resource-allocation-based search would provide stronger comparisons, although an exhaustive evaluation of all hyperparameter optimization methods is unnecessary.

  2. Validation fitness noise and adaptive selection bias are insufficiently analyzed.
    Validation sets range from 55 tasks on AppWorld to 512 tasks on Nemotron, and evaluation involves stochastic generation. Repeatedly making selection decisions from these scores may make member rankings sensitive to sampling noise. Repeated optimization against the same validation set may also introduce selection bias. The improvements on independent test sets provide meaningful evidence of generalization, so small validation sets alone do not establish overfitting. However, it remains unclear whether rankings are stable under repeated evaluation, whether selection thresholds are sufficiently large relative to measurement noise, and whether validation and test improvements diverge later in training. Targeted repeated evaluations or task-level resampling analyses would strengthen confidence in the selection mechanism.

  3. Independent population-level repetitions and uncertainty estimates are missing.
    PBPT members exchange training states and are therefore statistically dependent. Eight members should not be treated as eight independent repetitions of the method. Smoothed mean curves and averages over the final 20% of training illustrate trends but are insufficient to establish the stability of smaller performance differences. Two sources of uncertainty should be distinguished: measurement uncertainty arising from the sampled evaluation tasks, and training variability arising from rerunning the entire system. Appropriate task-level resampling can help estimate the former, while independent population runs are needed for the latter. Reporting only the standard deviation across members would not address population-level training variability.

  4. Improvements in population reliability and final model quality should be distinguished more clearly.
    Under tuned initialization on Enigmata, full PBPT improves average test accuracy from 75.40% to 77.59%, but the test accuracy of the member selected by validation performance increases only from 78.00% to 78.12%. These results more strongly support improved average performance and reduced failure risk than improved performance of the final selected model. Both objectives are legitimate, but practitioners often ultimately deploy a single model. The paper should report test performance under a consistent, predefined validation-based model selection protocol across all benchmarks and discuss this separately from population-average performance.

  5. The additional insights beyond existing PBT methods need to be strengthened.
    Selection, crossover, mutation, and decentralized execution all have precedents. The paper’s primary contribution is their adaptation to LLM RLVR and the accompanying systematic evaluation. Such a contribution can be valuable, but the paper should more explicitly identify the challenges specific to LLM post-training, which design choices address them, and which findings go beyond what would already be expected from PBT. The finding that selection is already effective under good initialization, while mutation becomes more important under poor initialization, is worth emphasizing. However, the main mechanism ablations are concentrated on Enigmata. Verifying a key finding on a tool-use benchmark would strengthen the evidence for its broader applicability.

Questions for authors
  1. Does PBPT retain its advantage when independent RLVR runs use the same initial configuration set? Could this comparison separate the benefits of initial configuration diversity from those of population interaction during training?

  2. What are the individual contributions of initial mutation, online mutation, and training-state transfer? Could the authors provide initial-mutation-only and online-mutation-only comparisons, or clarify precisely which conclusions the existing ablations support? Do the key findings also hold on a tool-use benchmark?

  3. How sensitive are selection decisions to fitness noise? For representative checkpoints, are member rankings and replacement decisions stable under repeated evaluation? How do the decision thresholds compare with the observed evaluation variability?

  4. Are the results reproducible across independent population runs? Could the authors report repetitions for representative settings, including both population-average performance and the performance of the final model selected using a validation-based protocol?

  5. How much of the advantage remains under fixed decoding configurations and inference budgets? This would help distinguish improvements from weight training from those due to evaluation-configuration optimization.

  6. What are the full computational costs and the trade-offs under a fixed total budget?
    How much do differences in evaluation workloads affect the comparisons? Under the same total budget, is it more beneficial to increase the number of members or to train each member longer?

5 comments

We sincerely thank you for your detailed review and constructive feedback. We appreciate your recognition that our work addresses an "important problem," presents a "well-developed system," and demonstrates "useful improvements in performance and training stability." We address your concerns and questions point by point below.

W1 and Q1: Initial Configuration Diversity

Following your suggestion, we ran an initial-mutation-only baseline on Enigmata, Nemotron, and AppWorld. The 8 learners start from the same configurations as PBPT, including its initial mutations, and then train independently without subsequent mutation or training-state transfer. All comparisons use Qwen3.5-4B with the same number of learners and per-learner training budget.

We follow the protocol in Table 1 of the revised manuscript, averaging evaluations over the final 20% of the 66-hour budget. The following table (Table 8 in Appendix B of the revised manuscript) reports mean test performance (%) across learners and for the learners with the highest and lowest validation means, respectively.

BenchmarkMethodAvg. testBest-val. testWorst-val. test
EnigmataPBPT (init. mut. only)75.8277.4370.86
EnigmataPBPT (full)77.5978.1277.03
NemotronPBPT (init. mut. only)45.8960.0136.69
NemotronPBPT (full)60.5762.0360.85
AppWorld (normal)PBPT (init. mut. only)75.6172.7766.67
AppWorld (normal)PBPT (full)84.7386.9080.44
AppWorld (challenge)PBPT (init. mut. only)52.6951.1447.91
AppWorld (challenge)PBPT (full)63.1864.2467.15

Full PBPT retains its advantage with matched initial configurations across all three benchmarks and all three reported measures. These results show that the gains cannot be explained by initial configuration diversity alone and that population decisions during training provide additional benefits.

Q2: Contributions of Initial Mutation, Online Mutation, and State Transfer

Following your suggestion, we tested initial-mutation-only and online-mutation-only variants of PBPT on Nemotron. The initial-mutation-only results are reported in our response to W1 and Q1 above. For the online-mutation-only variant, we disable initial mutation and training-state transfer, while retaining only mutations for underperforming learners. We use Qwen3.5-4B with the same number of learners and per-learner training budget. The following table reports test performance in the same way as our response to W1 and Q1.

MethodAvg. testBest-val. testWorst-val. test
PBPT (online mut. only)52.1957.0038.57
PBPT (full)60.5762.0360.85

Our manuscript also includes a state-transfer-only ablation on Enigmata. We call it "PBPT without mutation." See Section 5.3 and Figure 4 for details. Together, these comparisons show that none of the three isolated mechanisms consistently matches full PBPT, supporting the value of combining the three of them.

Regarding tool use, following your suggestion, we are extending the mechanism ablations to AppWorld. These experiments may not finish within the rebuttal period due to limited compute, but we will include the results in the next version of the manuscript.

W2 and Q3: Validation Noise and Selection Stability

Following your suggestion, we randomly sampled 50 Keep, 50 Perturb, and 50 Replace decisions from the Nemotron PBPT run in Section 5.1 of the revised manuscript. We sampled only events that pass the decision-eligibility checks. For each event, we re-evaluated the member's checkpoint, kept the other learners' scores unchanged, and replayed the decision. The following table reports the probability of each replayed decision conditional on the original decision.

Original decisionReplay: KeepReplay: PerturbReplay: Replace
Keep84%8%8%
Perturb54%30%16%
Replace44%10%46%

We repeated the same procedure on AppWorld, again sampling 50 events for each original decision.

Original decisionReplay: KeepReplay: PerturbReplay: Replace
Keep84%10%6%
Perturb46%32%22%
Replace40%12%48%

On both benchmarks, Keep decisions remain unchanged in 84% of replays. Most Perturb and Replace decisions either remain unchanged or become Keep, while switches between the two intervention types are less common. This pattern is consistent with our design of allowing learners to continue training unless their relative fitness warrants a configuration change or weight replacement.

In the Nemotron replays, the median absolute change in validation score is 0.88 percentage points, compared with median thresholds of 1.03 for mutation and 2.57 for replacement. On AppWorld, the median absolute change is 1.82 percentage points, compared with median thresholds of 1.89 for mutation and 4.73 for replacement. Thus, typical re-evaluation fluctuations are comparable to the mutation thresholds but substantially smaller than the replacement thresholds, consistent with our design of requiring stronger evidence to replace model weights than to explore a different configuration.

W3 and Q4: Independent Repetitions

We agree that evaluation uncertainty and training variability should be distinguished. Therefore, we repeated the complete PBPT experiment on Nemotron with a different random seed. We use Qwen3.5-4B with the same number of learners and per-learner training budget. The following table reports test performance in the same way as our response to W1 and Q1.

MethodAvg. testBest-val. testWorst-val. test
PBPT (original)60.5762.0360.85
PBPT (rerun)59.7660.0659.35

We can see that the original run and the rerun achieve similar performance. We view this consistency as evidence of PBPT's stability across independent repetitions.

W4 and Q4: Population Performance and Validation-Selected Models

We agree that improved population performance and improved performance of a validation-selected model are distinct claims. On Nemotron, using the same final-20% averaging protocol as above, the learners with the highest validation means achieve test accuracies of 62.03% in the PBPT run, compared with 56.81% for independent RLVR. These results show that PBPT's gains extend beyond the population mean to the performance of a learner selected using validation alone.

The gain for the best-validation learner is indeed small on Enigmata with the tuned initial configuration. Note that if independent RLVR already approaches the base model's attainable performance on a benchmark, there is limited room to improve the best learner. Therefore, PBPT may offer only a small improvement in the best learner while still improving average performance and the weaker learners, as seen on tuned Enigmata in Table 1 of the revised manuscript. With the suboptimal configuration, PBPT also achieves a much larger best-validation gain on Enigmata.

W5: Contribution Beyond Existing PBT Methods

Thank you for this suggestion. Following this feedback, we have fully rewritten Sections 3 and 4 in the revised manuscript to clarify PBPT's contributions and its relationship to prior PBT methods. Section 3 now explicitly contrasts PBPT with standard PBT and DexPBT, and Section 4 explains how our design addresses unique challenges in LLM post-training.

Specifically, in PBPT, we compare checkpoints at matched compute to account for configuration-dependent training costs. We require a larger fitness gap for weight replacement than for mutation, allowing configuration changes without prematurely discarding useful policies under noisy validation. We also allow mutation operators to jointly explore continuous and discrete RLVR settings, including the RL objective.

We are glad that you find the effectiveness of selection under good initialization interesting. We are extending the mechanism ablations to AppWorld, as described in our response to Q2 above.

Q5: Fixed Decoding Configurations and Inference Budgets

We would like to first clarify that our experiments already use a fixed total inference budget for all methods. Within each benchmark, RLVR and PBPT have the same limit on the total number of tokens generated per task.

PBPT may only change the inference temperature and, on tool-use benchmarks, the per-turn generation limit, but we keep the total token budget fixed to ensure a fair comparison.

How much of the improvement is attributed to inference settings is an interesting question. We reran PBPT on Nemotron with the inference temperature fixed at 0.6, the same value as RLVR. We use Qwen3.5-4B with the same number of learners and per-learner training budget. The following table reports test performance in the same way as our response to W1 and Q1.

MethodAvg. testBest-val. testWorst-val. test
RLVR49.2856.8144.48
PBPT60.5762.0360.85
PBPT (fixed inference)58.6059.4158.15

PBPT retains a clear advantage over RLVR even when the inference temperature is fixed although, evidently, inference config optimization does influence the final accuracy.

Ultimately, the resulting artifact of training consists of model weights, inference settings, and a tool-use harness. An effective optimization method should ideally optimize all three. PBPT naturally optimizes model weights and inference settings jointly, and in the future we envision extending it with agentic methods to evolve prompts and tool-use harnesses.

Q6: Computational Costs and Budget Allocation

Each method uses 32 B200 GPUs across 8 learners, with a nominal budget of 2,112 GPU-hours on Enigmata, Nemotron, and AppWorld, and 2,880 GPU-hours on Gaia2, as detailed in Appendix A.1.

Regarding evaluation, PBPT and independent RLVR checkpoints within each benchmark have the same limit on the total number of tokens generated per task. PBPT may change the evaluation temperature and, on tool-use tasks, the per-turn generation limit, but the total budget remains fixed. We have clarified this distinction in Appendix A.1 of the revised manuscript. Evaluation shares each learner's 4 B200 GPUs with training, and concurrent evaluation contributes to the reported training-time budget. Thus, a larger evaluation workload uses resources that would otherwise be available for training.

Under a fixed total budget, increasing the number of learners reduces the training time available to each learner. Our practical recommendation is to allocate enough time for each learner to approach a performance plateau, then use additional compute to increase population size. This recommendation is motivated by Figure 2 of the revised manuscript, where extending training yields larger gains before the curves plateau than afterwards. We have not systematically compared different allocations between population size and training duration under a fixed total budget.

We hope our responses address most of your concerns, and we would appreciate your reconsideration of our work in light of the new results and manuscript revisions.

Comments on the paper

1 comment

We sincerely thank all reviewers for their time and efforts. We appreciate the positive assessment of our presentation as "Clear and easy to follow" (3UxB, yGgL, 6YSG). We also appreciate the recognition that our work addresses an "important problem" (3UxB) in "a valuable optimization setting" (yGgL).

A shared concern from 3UxB and yGgL is that our contributions beyond existing PBT methods were not sufficiently clear. Following this feedback, we have fully rewritten Sections 3 and 4 in the revised manuscript to clarify PBPT's contributions and its relationship to prior work. Section 3 now explicitly compares PBPT with prior work, and Section 4 explains how our design addresses unique challenges in LLM post-training.

We have also conducted extensive additional experiments in response to the reviewers' specific suggestions. We report the results in Appendix B of the revised manuscript and in our responses to the individual questions below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.