- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how the gold-reward gain from optimizing a proxy reward model depends jointly on the number of preference comparisons M and the KL budget K. In a stylized model, gold rewards are i.i.d. Gaussian across responses. The paper conducts extensive empirical evaluations using a real-world annotation setup. However, the theory does not explain the experiments well.
Critical strengths
- The question is useful. Existing work left the dependence on preference-data size unexplained, and a rule for setting the KL budget from M would directly guide practice.
- The theory is clean. The experiments are carefully controlled, with a shared comparison stream, shared initialization and shared evaluation pools.
Critical weaknesses
The theory does not explain the experiments. Rewards are i.i.d. across responses, so comparisons carry no information about responses outside them. The experiments evaluate on 128 new prompts with 8,192 freshly sampled responses each, none of which appear in the training data. Under the model, each of these responses has conditional mean , so the predicted gain is zero. The observed gain must come from the reward model generalizing to unseen text, which the model does not capture.
Questions for authors
- Your model predicts zero gain on unseen responses. Which part of the theory do you expect to transfer to a proxy that generalizes across responses? Can it be extended to rewards that share features?
- Can you choose N on validation prompts, report on separate test prompts, and show how the maximizing KL scales with M? Does the peak reach the boundary N = 8,192 for large M?
- Can you vary the number of prompts and the number of comparisons per prompt independently? Are the ten models separate runs or checkpoints of one run?
3 comments
Rebuttal to Reviewer 4YWK (Part 2/2)
Can you vary the number of prompts and the number of comparisons per prompt independently? Are the ten models separate runs or checkpoints of one run?
Yes. We performed a full ablation with the 1.7B model, independently varying training prompts Q{32,64,128,534,1000} and comparisons per prompt C{10,25,50,100,150,300}.
For each fixed C, we fit held-out BoN reward gain using :
| C | 10 | 25 | 50 | 100 | 150 | 300 |
|---|---|---|---|---|---|---|
| 0.987 | 0.976 | 0.958 | 0.974 | 0.967 | 0.999 |
These results show that our proposed functional form continues to describe prompt scaling across comparison depths.
The ten models are checkpoints of one training run per model-size/noise setting. Our preliminary hyperparameter sweeps supported one epoch of training with a constant learning rate of 10-5. We also found that a broad range of learning rates worked well, consistent with InstructGPT [1]. Each prompt’s comparison group is processed once, so later checkpoints add new prompts without revisiting earlier ones. This makes the scaling study computationally feasible and follows the single-epoch, fixed-learning-rate approach to capture multiple checkpoints which is common in the literature [2,3]. We will clarify this protocol in the revision.
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
[3] Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for llms. In ICLR, 2026.
Thanks for your response. It solved my concerns. I have raised my rating.
Rebuttal to Reviewer 4YWK (Part 1/2)
We thank the reviewer for the constructive questions and for their patience while we ran the requested experiments. We agree that the distinction between the theoretical selection problem and generalization via a learned reward model needed further clarification. We address this limitation below and report additional empirical findings addressing the reviewer’s questions.
It is true that in our original model, the gains come from overlapping answers in the admissible set. This is a stylized model. In reality, generalization gains come from correlations across prompts. As explained below, we can capture this mechanism through a simple extension of our model and analysis.
In our original model, responses outside the set of compared responses V(E) retain conditional mean μ, so expected gain comes from A∩V(E), whose expected size is p∣V(E)∣. A fresh prompt in this theoretical model draws a new admissible set from the same response universe, its support is not in general disjoint from the compared responses.
Now, we extend our model to correlated Gaussian rewards with covariance Σ and common marginal variance σ2 . The off-diagonal entries of Σ encode shared reward structure, allowing comparisons to raise unseen responses’ posterior means above μ and thereby capture generalization. Under the same offline comparison and admissibility assumptions, we obtain
\qquad \kappa := \frac{\lambda_{\max}(\Sigma)}{\sigma^2}.$$ **This preserves the $\sqrt{\log M}$ upper-bound scaling** (assuming $\kappa$ is bounded independently of M). We include the proof in the revision in Appendix B and thank the reviewer for highlighting this useful extension. > Can it be extended to rewards that share features? Yes. Consider linearly parameterized rewards $$r_\theta(x,y)=\mu+\phi(x,y)^\top\theta, \qquad \theta\sim\mathcal N(0,C),$$ where $x$ is the prompt, $y$ is the completion, and $\phi$ is a fixed feature representation, potentially nonlinear in the text. Comparisons inform the shared parameter $\theta$, enabling generalization to unseen responses. Writing $\Phi$ for the feature matrix gives reward covariance $\Sigma = \Phi C \Phi^{\top}$. If marginal variances are bounded by $\sigma^2$, under the same offline comparison and admissibility assumptions, $$\Delta \leq \sigma\sqrt{2\min\left\\{K,\log(1+\kappa pM)\right\\}}, \qquad \kappa := \frac{\lVert\Phi C^{1/2}\rVert_{\mathrm{op}}^2}{\sigma^2}.$$ Thus, the bounded operator-norm assumption $\|\Phi C^{1/2}\|_{\mathrm{op}}^2\le \kappa_0\sigma^2$, for a constant $\kappa_0$ independent of M, preserves the scaling. We include the proof in the revised paper and is similar to the previous one. > Can you choose N on validation prompts, report on separate test prompts, and show how the maximizing KL scales with M? Does the peak reach the boundary N = 8,192 for large M? We extended BoN evaluation from **8,192 to 16,384 responses per prompt**. We split the held-out prompts into 25% validation prompts and 75% separate test prompts. For each training budget M, we select the reward-maximizing N on the validation prompts and report performance on the test prompts. We plot in Appendix D the selected $N^\star(M)$, which is analytically related to the BoN KL: $$K^\star(M)=\log N^\star(M)-\frac{N^\star(M)-1}{N^\star(M)}.$$ For the 1.7B noiseless model shown in Appendix D, the selected N generally increases with M, and the boundary N=16,384 is reached **only at the largest budget, M=300,000**. **The square-root-logarithmic trend continues to hold on separately evaluated test gains**, with $R^2$=0.986.