- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
An evolutionary algorithm for hyper parameter optimization during post training.
Critical strengths
The use of mutation and crossover and the weight sharing during post-training are creative and interesting.
Critical weaknesses
None
Questions for authors
Does the mutation ablation include crossover or not? Generally, I would have liked to see more on the efficacy of the crossover mechanism. I'd also be interested to learn more about how you set the various thresholds and whether they are even needed. For example, you could choose always to copy weights in all tournaments, and always to perturb anyway. Same goes for the evaluation schedule spacing. Did you test starting from human presets (with some perturbations) versus an all random starting point?
3 comments
Thank you for the explanations and revisions. The paper looks much stronger for it.
Thank you for your recognition and support for our work!
We sincerely thank you for your support and positive feedback. We are glad that you find our use of mutation, crossover, and weight sharing during post-training "creative and interesting." We address your questions below.
Crossover and the Mutation Ablation
We appreciate your interest in the crossover mechanism.
In the mutation ablation, we disable initial and subsequent mutation, so all learners retain the same hyperparameter configuration. Crossover remains enabled, but combining identical configurations has no effect.
In full PBPT, crossover plays a key role. Configuration diversity is important for exploring the hyperparameter search space effectively. Simply copying the donor's entire configuration at each replacement would reduce this diversity. Crossover combines donor and recipient hyperparameters, which allows successful settings to spread while retaining variation across learners.
Decision and Evaluation
Weight replacement threshold (why not always copy weights?) In our early experiments, we found that checkpoints with a temporary fitness advantage did not necessarily lead to the best final policies. For example, mutation towards a more aggressive configuration, such as a higher learning rate, could yield rapid early gains but lower final performance. Requiring a sufficient fitness gap before copying weights reduces the risk of such checkpoints dominating the population prematurely.
Mutation threshold (why not always perturb anyway?) Always perturbing would make it difficult to retain a good configuration long enough to benefit from it and propagate it through selection. The mutation threshold allows a learner to continue training with its current configuration when its fitness gap is small.
Winner and loser proportions. We divide the population approximately equally into winners, keepers, and losers. For our 8-member population, we use a 3/3/2 split, allocating the extra member to the winner set to increase donor diversity while keeping the loser set smaller to reduce unnecessary weight replacements. These proportions were chosen heuristically and can be considered hyperparameters of the method.
Warm-up and cooldown. Warm-up and cooldown are motivated by our prior experience with PBT. A newly mutated configuration needs time to influence training before its effectiveness can be assessed. Warm-up provides this time at the start of training, while cooldown does so after a configuration change.
Evaluation schedule. Evaluating too frequently consumes compute that would otherwise be available for training. Evaluating too infrequently provides less feedback for population decisions and too few data points in plots such as Figure 2. We choose the evaluation intervals to balance these considerations.
Initial Configurations
We evaluated a suboptimal initial configuration on Enigmata in addition to the human-tuned configurations used in our main experiments. See the PBPT and RLVR results from Table 1 of the revised manuscript below, averaged over the final 20% of the training budget. The following table reports mean test performance (%) across learners and for the learners with the highest and lowest validation means, respectively.
PBPT's advantage over independent RLVR is much larger under the suboptimal initialization. PBPT approaches its performance from the tuned initialization, with a small remaining gap, whereas independent RLVR degrades severely. The suboptimal configuration was still chosen by a human engineer to be intuitively reasonable, but without iterative tuning. We have not evaluated PBPT with its entire population initialized randomly from the search space. Our search space is deliberately broad, so such sampling can produce configurations that make little progress within the training budget, such as learning rates below 10−7. We would expect mutations to give PBPT an even larger advantage over fixed-configuration RLVR in that setting, but we consider the comparison against plausible initial configurations more informative.
We hope our responses address your questions, and we thank you again for your support of our work.