acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Scaling Laws for Reward Optimization

Abstract

Scaling laws for optimization against reward models in AI alignment have pinned down how performance depends on optimization effort–measured by a KL-divergence budget relative to a reference policy. Beyond a certain budget, over-optimization (or reward hacking) can arise: because we optimize against a proxy reward model (distinct from true rewards), performance can plateau or degrade. Naturally, the proxy reward's accuracy depends on how much preference data (often in the form of pairwise comparisons) was used to train it. However, existing research does not cleanly identify how performance jointly scales with the amount of training data and the divergence budget. Our main contribution is to provide an empirically accurate and theoretically grounded scaling law in such context. Performance roughly scales as , where is the number of comparisons in training data and is the policy's divergence budget. We develop an information-theoretic model to establish this upper bound and prove it is tightly achievable through a constructive procedure. Informed by this, we conduct extensive empirical evaluations using a real-world annotation setup, whereby a large 70B gold reward model generates feedback data and proxy reward models are trained from less capable models (0.6B to 4B). Our scaling law provides an excellent fit (R2 from 97% to 99%), outperforms alternative specifications, and remains robust across model sizes, noise, and optimization procedures (best-of- or policy tilting). Our evidence suggests that reward optimization is analogous to a surprisingly simple selection task: choosing from a sequence of IID Gaussian random variables using noisy preference feedback.

Primary area:
reinforcement learning
Keywords:
Reward optimization, Reward hacking, Information Theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_4YWKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the gold-reward gain from optimizing a proxy reward model depends jointly on the number of preference comparisons M and the KL budget K. In a stylized model, gold rewards are i.i.d. Gaussian across responses. The paper conducts extensive empirical evaluations using a real-world annotation setup. However, the theory does not explain the experiments well.

Critical strengths
  1. The question is useful. Existing work left the dependence on preference-data size unexplained, and a rule for setting the KL budget from M would directly guide practice.
  2. The theory is clean. The experiments are carefully controlled, with a shared comparison stream, shared initialization and shared evaluation pools.
Critical weaknesses

The theory does not explain the experiments. Rewards are i.i.d. across responses, so comparisons carry no information about responses outside them. The experiments evaluate on 128 new prompts with 8,192 freshly sampled responses each, none of which appear in the training data. Under the model, each of these responses has conditional mean μ\mu, so the predicted gain is zero. The observed gain must come from the reward model generalizing to unseen text, which the model does not capture.

Questions for authors
  1. Your model predicts zero gain on unseen responses. Which part of the theory do you expect to transfer to a proxy that generalizes across responses? Can it be extended to rewards that share features?
  2. Can you choose N on validation prompts, report on separate test prompts, and show how the maximizing KL scales with M? Does the peak reach the boundary N = 8,192 for large M?
  3. Can you vary the number of prompts and the number of comparisons per prompt independently? Are the ten models separate runs or checkpoints of one run?
3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 4YWK (Part 1/2)

We thank the reviewer for the constructive questions and for their patience while we ran the requested experiments. We agree that the distinction between the theoretical selection problem and generalization via a learned reward model needed further clarification. We address this limitation below and report additional empirical findings addressing the reviewer’s questions.

The theory does not explain the experiments. Rewards are i.i.d. across responses, so comparisons carry no information about responses outside them. The experiments evaluate on 128 new prompts with 8,192 freshly sampled responses each, none of which appear in the training data. Under the model, each of these responses has conditional mean , so the predicted gain is zero. The observed gain must come from the reward model generalizing to unseen text, which the model does not capture.

It is true that in our original model, the gains come from overlapping answers in the admissible set. This is a stylized model. In reality, generalization gains come from correlations across prompts. As explained below, we can capture this mechanism through a simple extension of our model and analysis.

In our original model, responses outside the set of compared responses V(E)V(E) retain conditional mean μ\mu, so expected gain comes from A∩V(E)A\cap V(E), whose expected size is p∣V(E)∣p|V(E)|. A fresh prompt in this theoretical model draws a new admissible set from the same response universe, its support is not in general disjoint from the compared responses.

Now, we extend our model to correlated Gaussian rewards with covariance Σ\Sigma and common marginal variance σ2\sigma^2 . The off-diagonal entries of Σ\Sigma encode shared reward structure, allowing comparisons to raise unseen responses’ posterior means above μ\mu and thereby capture generalization. Under the same offline comparison and admissibility assumptions, we obtain

\qquad \kappa := \frac{\lambda_{\max}(\Sigma)}{\sigma^2}.$$ **This preserves the $\sqrt{\log M}$ upper-bound scaling** (assuming $\kappa$ is bounded independently of M). We include the proof in the revision in Appendix B and thank the reviewer for highlighting this useful extension. > Can it be extended to rewards that share features? Yes. Consider linearly parameterized rewards $$r_\theta(x,y)=\mu+\phi(x,y)^\top\theta, \qquad \theta\sim\mathcal N(0,C),$$ where $x$ is the prompt, $y$ is the completion, and $\phi$ is a fixed feature representation, potentially nonlinear in the text. Comparisons inform the shared parameter $\theta$, enabling generalization to unseen responses. Writing $\Phi$ for the feature matrix gives reward covariance $\Sigma = \Phi C \Phi^{\top}$. If marginal variances are bounded by $\sigma^2$, under the same offline comparison and admissibility assumptions, $$\Delta \leq \sigma\sqrt{2\min\left\\{K,\log(1+\kappa pM)\right\\}}, \qquad \kappa := \frac{\lVert\Phi C^{1/2}\rVert_{\mathrm{op}}^2}{\sigma^2}.$$ Thus, the bounded operator-norm assumption $\|\Phi C^{1/2}\|_{\mathrm{op}}^2\le \kappa_0\sigma^2$, for a constant $\kappa_0$ independent of M, preserves the scaling. We include the proof in the revised paper and is similar to the previous one. > Can you choose N on validation prompts, report on separate test prompts, and show how the maximizing KL scales with M? Does the peak reach the boundary N = 8,192 for large M? We extended BoN evaluation from **8,192 to 16,384 responses per prompt**. We split the held-out prompts into 25% validation prompts and 75% separate test prompts. For each training budget M, we select the reward-maximizing N on the validation prompts and report performance on the test prompts. We plot in Appendix D the selected $N^\star(M)$, which is analytically related to the BoN KL: $$K^\star(M)=\log N^\star(M)-\frac{N^\star(M)-1}{N^\star(M)}.$$ For the 1.7B noiseless model shown in Appendix D, the selected N generally increases with M, and the boundary N=16,384 is reached **only at the largest budget, M=300,000**. **The square-root-logarithmic trend continues to hold on separately evaluated test gains**, with $R^2$=0.986.
AuthorsOpenReview ↗

Rebuttal to Reviewer 4YWK (Part 2/2)

Can you vary the number of prompts and the number of comparisons per prompt independently? Are the ten models separate runs or checkpoints of one run?

Yes. We performed a full ablation with the 1.7B model, independently varying training prompts Q{32,64,128,534,1000} and comparisons per prompt C{10,25,50,100,150,300}.

For each fixed C, we fit held-out BoN reward gain using aC+bClog⁡(QC)a_C+b_C \sqrt{\log (Q_C)}:

C102550100150300
R2R^20.9870.9760.9580.9740.9670.999

These results show that our proposed functional form continues to describe prompt scaling across comparison depths.

The ten models are checkpoints of one training run per model-size/noise setting. Our preliminary hyperparameter sweeps supported one epoch of training with a constant learning rate of 10-5. We also found that a broad range of learning rates worked well, consistent with InstructGPT [1]. Each prompt’s comparison group is processed once, so later checkpoints add new prompts without revisiting earlier ones. This makes the scaling study computationally feasible and follows the single-epoch, fixed-learning-rate approach to capture multiple checkpoints which is common in the literature [2,3]. We will clarify this protocol in the revision.

[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022.

[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.

[3] Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for llms. In ICLR, 2026.

Thanks for your response. It solved my concerns. I have raised my rating.

Reviewer_aKNBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a theoretical formulation of reward hacking and empirically validates the proposed theory.

Critical strengths
  1. This paper builds a reasonable theoretical result for reward overoptimization.
  2. The experiment results can well verify the theoretical results.
Critical weaknesses
  1. Which insight can be benefited from these theoretical results for building a better reward model?
  2. What is the meaning of "choosing among a sequence of IID Gaussian random variables from noisy preference data."
Questions for authors
  1. Are these results are suitable to different reward model ,such as Bayesian reward model?
  2. Are these results related to LLM parameters?
1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer aKNB

We thank the reviewer for the positive assessment of the theoretical and experimental results, and for their patience while we conducted experiments to strengthen the paper. We clarify below the practical implications, the interpretation of the Gaussian model, and the scope of the results.

Which insight can be benefited from these theoretical results for building a better reward model?

Our main practical contribution is a scaling law that guides what true reward improvement to expect as a function of preference-dataset size. In the comparison-limited regime, the proposed form a+blog⁡Ma+b\sqrt{\log M} captures the diminishing returns from additional preference data. After calibrating the coefficients for a given setting on small-scale data, practitioners can estimate the benefit of expanding their dataset and plan annotation budgets for a desired reward improvement. Our experiments show that our scaling law is robust across different proxy model sizes, preference-noise settings, and optimization methods. We will make this practical implication clearer in the revision.

Are these results are suitable to different reward model ,such as Bayesian reward model?

Yes, the upper bounds apply to any reward-modeling procedure under the stated reward and information assumptions, including Bayesian procedures. They constrain the information available from the MM comparisons, rather than requiring a particular architecture or training algorithm. A Bayesian reward model may maintain a posterior over rewards and use it to guide selection. The resulting policy remains subject to the same bound when it satisfies the KL constraint.

Indeed, our original setup already admits a Bayesian interpretation: Gaussian latent rewards form the prior, and deterministic or Bradley-Terry comparisons provide the likelihood. The posterior does not need to remain Gaussian. Our correlated-Gaussian extension also covers Bayesian shared-feature models with Gaussian parameter priors, preserving the same scaling.

Are these results related to LLM parameters?

Yes, through the resulting policy: RL updates the LLM’s parameters, changing its response distribution and KL divergence K, which our theory relates to achievable reward improvement. Parameter count itself is not an explicit variable in our theorem.

Our main contribution is a scaling law with respect to preference-dataset size M, jointly with optimization budget K. Earlier work by Gao et al.[1] characterized scaling with reward-model parameter count. We complement this by theoretically grounding the dependence on preference data quantity and empirically testing it across 0.6B, 1.7B, and 4B reward proxies.

We hope our clarifications address your questions given your initial positive assessment.

[1] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835–10866. PMLR, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.