- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper systematically studies reward model overoptimization in iterated RLHF, focusing on three design choices: how preference data is combined across iterations, how reward models are combined, and how policies are initialized. Using simulated preferences, it finds that aggregating historical preference data and resetting the policy to its SFT checkpoint generally improve robustness in the evaluated setting. Experiments primarily use a 410M policy, with selected comparisons at 1.4B, additional reward-model sizes, and supplementary evaluations.
Critical strengths
The paper provides a structured comparison of consequential design choices in iterated RLHF. Studying data management, reward-model combination, and policy initialization within a common framework helps identify how these choices affect overoptimization across iterations.
The experimental controls strengthen the analysis. Multiple random seeds, label-noise experiments, and a larger-data single-iteration baseline provide evidence that the observed benefits are not explained solely by additional preference data.
Critical weaknesses
The evidence supporting general practical recommendations remains limited. The full study uses a 410M policy, while the 1.4B experiments cover selected comparisons; proxy reward models are only 70M and 160M, and the main preference collection repeatedly uses 1,000 instructions. Importantly, continuation becomes competitive when reward-model capacity increases, suggesting that the recommendation to reset policies is regime-dependent. The TL;DR evaluation also reports declining gold reward within every iteration, so it does not clearly validate improved policy quality on a second task. These limitations reduce my confidence that the recommendations transfer to more capable models and broader training settings.
The policy-initialization comparison does not sufficiently separate the benefits of resetting from the consequences of inheriting an already overoptimized checkpoint. Under the fixed optimization budget, continuation can begin subsequent rounds from a substantially degraded policy. This demonstrates the value of resetting after collapse, but leaves unclear whether resetting remains preferable when earlier feedback collection, stronger regularization, or appropriate stopping prevents collapse. Establishing this distinction would make the practical insight more consequential.
Questions for authors
- Does the advantage of resetting persist when continuation uses earlier stopping or more frequent reward-model updates to avoid collapse in the first iteration?
- Given that continuation becomes competitive with the 160M reward model, what evidence supports recommending resets beyond the capacity-limited regime studied here?
- How should the cross-task conclusions be interpreted when gold reward decreases within every iteration on TL;DR? Can the authors distinguish improved reward-model robustness from policy-optimization failure?
4 comments
Official Comment by Authors Part 2/2
3. Resets beyond the capacity-limited regime (Question 2)
The finding that Continue becomes competitive with larger reward models is one of the contributions of our paper and is explicitly stated in the abstract. It does not make Continue preferable though, since with all other design choices held fixed, From SFT achieves a similar or higher gold score than Continue in every setting (see the table below, or equivalently Table 11, Figure 5 and Figure 6 in the revised paper). Resetting is the safer and more robust option, since in absence of a gold reward signal, practitioners cannot determine whether the current policy has started to overoptimise. We therefore recommend the option that performs at least as well in all these settings and substantially better whenever the policy has overoptimised.
| Setting | From SFT | Continue |
|---|---|---|
| 410M policy, 70M reward model | 0.45 ± 0.07 | 0.20 ± 0.18 |
| 410M policy, 160M reward model | 0.57 ± 0.02 | 0.53 ± 0.06 |
| 1.4B policy, 70M reward model | 0.47 ± 0.07 | −0.22 ± 0.20 |
| 2.8B policy, 70M reward model | 0.57 ± 0.05 | 0.33 ± 0.14 |
Gold score at the end of iteration 4 on the same preference data (mean ± one standard error over seeds).
4. TL;DR (Weakness 1, Question 3)
Our claim regarding TL;DR concerns reward model overoptimisation, i.e., the gap between proxy and gold reward, which decreases across iterations as observed on AlpacaFarm.
After the submission, we found a mismatch in the response length between training and evaluation in the setup that produced Figure 22. We have corrected this in the revision and adopted the standard PPO settings for this task (Huang et al., 2024). Summarisation is also a more challenging task for learning a reward model than AlpacaFarm, since the reward model has to judge a summary against a long post. We therefore use a 410M reward model and 4,000 preference pairs per iteration. With this setup we obtain the following results (see Table 9 and the updated Figure 22 of the revised paper):
- PPO on the gold reward model itself raises the gold reward steadily, so policy optimisation does succeed on this task.
- With Concatenate + From SFT, the gold reward improves consistently across iterations, ends iteration 4 above the SFT policy and approaches the level reached by directly optimising the gold reward.
- The gap between proxy and gold narrows across iterations, confirming the benefits of iterated RLHF in terms of reducing reward model overoptimisation.
Regarding Question 3, this confirms that policy optimisation is effective on this task, and the improvements across iterations result from the reward model becoming more robust. We have replaced Figure 22 and Appendix F.8 with the new results in the revision. In Appendix F.8, we also provide the full details of this setup.
Once again, we thank the reviewer for their time and effort. If our clarifications and additional results address the concerns and questions raised, we would kindly ask the reviewer to consider raising their score. We would, of course, be happy to answer any further questions.
References:
Gao, L., et al. (2023). Scaling Laws for Reward Model Overoptimization. In Proceedings of the 40th International Conference on Machine Learning.
Coste, T., et al. (2024). Reward Model Ensembles Help Mitigate Overoptimization. In The Twelfth International Conference on Learning Representations.
Eisenstein, J., et al. (2024). Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking. In First Conference on Language Modeling.
Moskovitz, T., et al. (2024). Confronting Reward Model Overoptimization with Constrained RLHF. In The Twelfth International Conference on Learning Representations.
Huang, S., et al. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. In First Conference on Language Modeling.
Response to authors
Thanks to the authors for providing additional experiments. Overall, the additional evidence increases my confidence in the reported findings, but does not sufficiently change my assessment of the contribution’s incremental insight and practical generality (e.g., my concern about the practical recommendation to reset policies remains partially addressed. Starting continuation from a non-overoptimized checkpoint does not establish whether resetting remains preferable when subsequent rounds use more frequent feedback collection or appropriate stopping to limit overoptimization). I therefore maintain my original rating of weak rejection.
Response by authors to remaining reviewer concern
We thank the reviewer for engaging in the discussion and are glad that our rebuttal and the additional evidence have increased their confidence in our the findings.
Regarding the remaining point, we would like to point out that the requested experiment is in fact already included in the revision. On TL;DR (Appendix F.8 and Table 10) both strategies use a significantly larger KL penalty than on AlpacaFarm, and policy optimisation is stopped in every round once the KL to the SFT policy exceeds 300 nats. Despite early stopping being triggered for Continue several times, it keeps overoptimising and underperforms the SFT policy and more strongly underperforms resetting (on average, and in 7 of 8 seeds).
As already mentioned in our response, it is also unclear what an appropriate stopping rule would be based on. In practice the gold reward is unknown, the proxy reward keeps increasing while the policy overoptimises, and the KL at which the highest gold reward is reached varies across iterations. There is therefore no signal for when to stop or when to collect feedback earlier. Resetting requires no such choice and achieves a similar or higher gold reward than Continue in every setting we have tested.
Please let us know if any other concerns remain, as we are happy to address them.
Official Comment by Authors Part 1/2
We thank the reviewer for their comments and their time and effort spent on reviewing our work. We address each of the raised weaknesses and questions below and kindly ask the reviewer to reconsider their score in light of our response.
1. Generality of recommendations (Weakness 1)
We emphasise that our empirical scale in terms of policy as well as reward model size is consistent with the established literature on reward model overoptimisation (Gao et al., 2023; Coste et al., 2024; Eisenstein et al., 2024; Moskovitz et al., 2024) and we have run additional experiments scaling further. In particular, we have run experiments with a 2.8B policy to confirm that the ordering of design choices at the end of iteration 4 remains unchanged at all policy sizes (see added results in Table 8 of the revised paper, and below). We also note that Gao et al. (2023) find no clear dependence of reward model overoptimisation on the policy size which justifies our choices.
Gold score at the end of iteration 4 (mean ± one standard error over seeds).
We have also run additional experiments on TL;DR further scaling the reward model size to 410M and scaled the number of collected preference pairs per iteration to 4k. We have improved the optimisation setup on TL;DR and observe consistent results. With Concatenate + From SFT the gold score improves across iterations relative to the SFT policy, and in iteration 4 it comes close to the improvement of +0.41 obtained by running PPO directly on the gold reward model (see added results in Table 9 of the revised paper, and below). As on AlpacaFarm, Concatenate outperforms Take last in terms of the gold reward (in 7 of 8 seeds, also see Table 10 in the revision). We also note that our dataset size ablation in Appendix F.3 is already based on 4k prompts, and a single iteration on this larger dataset still overoptimises more than the iterated setting.
Gold score at the end of each iteration (mean ± one standard error over seeds).
In our opinion, the already established consistency with prior literature in the field, our additional results for larger policies / reward models / preference datasets, and the justification regarding policy scaling address this concern.
2. Resetting versus inheriting an overoptimised checkpoint (Weakness 2 & Question 1)
Our results show that the advantage of resetting persists even when the previous iteration has not yet overoptimised. We have investigated Continue from a non-overoptimised starting point in three different settings (different policy sizes and reward models). In none of them does it outperform resetting, and with 70M reward models it eventually still overoptimises and underperforms resetting. This shows that the degradation of Continue is not only due to inheriting a bad checkpoint. Instead, it is caused by progressively extending optimisation, while resetting constraints the KL divergence across iterations (see Remark A.6).
We point out that stronger regularisation and earlier stopping cannot resolve this. Continue is already regularised in the same way as resetting, that is, the KL penalty is computed for both against the SFT policy, with the same coefficient, so Continue does not degrade because it lacks an anchor to the SFT policy. A higher KL penalty (Gao et al., 2023) or smaller learning rate only slows down the optimisation but does not prevent overoptimisation. Instead, we investigate the complementary question of how reward models can become sufficiently robust to support longer optimisation without themselves being overoptimised. This is important because there is no clear signal for early stopping: the gold reward is unknown, the proxy reward keeps increasing, and the KL value at which the highest gold score is achieved varies across iterations and is itself unknown.