acceptodds
Under review as a conference paper at ICLR 2027

What Does Static Reward-Model Evaluation Identify About Downstream Optimization?

Abstract

In reinforcement learning from human feedback (RLHF), static evaluations are commonly used to select reward models (RM) before policy optimization. We ask when such evaluations are sufficient to determine which RM yields a better policy. The question matters because training a policy for every candidate is expensive, while RMs with similar static accuracy are known to produce very different policies. Two RMs can rank every response identically and have the same reward variance, yet one improves the policy while the other makes it worse. Ranking and variance indicate how strongly an RM prefers some responses over others, not whether those preferences agree with the target's. The question is therefore not how many scores to observe, but whether they are taken on responses that the target itself distinguishes. We formalize this for reward models that are linear in a fixed set of features. Under a norm-bounded linear reward class, we derive the sharp range of initial effects consistent with exact score observations, showing that the pairwise comparison can be identified without recovering every reward score. This local comparison need not persist as the policy distribution changes. We derive sufficient conditions under which it remains identified after a finite policy change, including at matched KL distance. Experiments with pretrained RM scores show that partial observations can identify the comparison after finite optimization or leave both signs attainable. Static evaluation can thus guide RM selection, but only when it observes what the target cares about, and only for limited optimization.

Primary area:
reinforcement learning
Keywords:
reward models, RLHF, static evaluation, downstream optimization, partial identification, KL-regularized optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2wjoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how static reward model evaluations are connected to downstream policy optimization. Interestingly, they show that the initial improvement of the policy depends on the covariance of the proxy reward model and the target reward model. They then show that the responses that the reward model is measured on can matter most for uncovering this initial reward model optimization effect. In addition, they study how finite policy optimization affects reward model comparison. Most of this work is theory driven, but they also provide a variety of real reward modeling experiments.

Critical strengths
  1. RQ1 is a novel and useful finding.
  2. Although RQ2 and RQ3 are not that interesting to me, they do provide a detailed extension and further study of the ideas of the paper, making it a complete study.
  3. This paper provides both theoretical results and empirical analysis.
Critical weaknesses
  1. To me the story of the paper is not that clear (although the punchlines are). Although I am very excited about the results, the paper is still hard to follow.
  2. RQ1 seems to be the main contribution of the paper. It is an interesting finding. Otherwise, I do not find RQ2 or RQ3 to be too informative.
Questions for authors
  1. Can you explain the setting of Figure 2a? Is this online policy optimization assuming the same responses are sampled from the policy for each RM?
  2. Why does the policy score suddenly jump late in training in Figure 3? This is very strange.
2 comments

We sincerely thank Reviewer 2wjo for recognizing RQ1 as a novel and useful finding, and for valuing the paper’s combination of theoretical results and empirical analysis. We address each question and concern below.

W1: Story and presentation

RQ1 establishes the starting point: even complete response ordering together with equal reward variance can leave the sign of the initial target utility change undetermined.

Additionally, we have revised the manuscript accordingly and uploaded a new version. In Section 3.2, we moved the three response example into the main text. In Section 3.3, before Theorem 3.2, we now explain what information must be identified and how the notation maps to the empirical construction. In Section 4.4, before Theorem 4.3, we added the intuition for why matched KL requires comparison over the full feasible reward sets rather than at one common optimization strength.

These are presentation changes; the assumptions and theoretical results are unchanged.

W2: Why RQ2 and RQ3 are needed

RQ1 shows that ordering and reward variance can be insufficient, but it leaves open the question of what static information is sufficient. RQ2 answers this. For the specified linear reward class, Theorem 3.2 gives the necessary and sufficient identification condition, Theorem 3.3 gives the sharp identified interval when point identification fails, and Corollary 3.4 shows that the pairwise RM comparison can be identified even when neither individual local effect is point identified.

A second issue remains after the local comparison is identified: policy optimization changes the response distribution, so the local ordering need not persist. RQ3 studies this step. It separates the uncertainty already present in the static observations from the additional effect of policy movement, and gives finite comparison guarantees over the feasible reward sets, including at matched KL. The reversal constructions show why the local result alone is not enough for the finite optimization question.

So the role of RQ2 and RQ3 is to answer the two questions left open by RQ1: what static information resolves this ambiguity, and whether the resulting conclusion still holds after the policy moves.

Q1: Setting of Figure 2a

Figure 2a is not online policy optimization. It uses one fixed empirical response bank with 32 responses for each of 128 prompts, generated once and reused for rewards A–C.

For each prompt, Best-of-NN samples NN response indices uniformly with replacement from this bank and selects the highest-ranked sampled response. Rewards A–C have the same complete ordering and use the same fixed tie-breaking rule, so they select the same response from every common candidate set. The plotted curve is the exact expected target gain on this bank, computed from the rank-selection probabilities in Appendix B.2.

Figure 2a therefore isolates the effect of ranking: identical orderings give identical Best of NN outcomes. Figure 2b uses exact Gibbs optimization on the same bank, where reward magnitudes affect response probabilities and lead to different target gains at matched KL.

Q2: Why does the policy score suddenly jump late in training in Figure 3?

Figure 3 evaluates held-out utility every 100 updates, so the behavior between updates 1,200 and 1,300 is not shown.

  1. We checked the training and evaluation setup over this interval. The held-out prompts, sampling settings, seed rule, learning rate, and KL coefficient were unchanged.

  2. We resumed from the update-1,200 checkpoint and evaluated every 10 updates through 1,300.

UpdateA gainB gainA−BA-BA sequence log-ratioB sequence log-ratio
11000.0022786210.003071451−0.0007928296.8435555.084101
12000.0021978950.002325078-0.0001271837.4437177.494322
12100.0022863400.002518624-0.0002322847.4797097.593199
12200.0020917720.002411953-0.0003201817.6833977.611277
12300.0021746650.002901481-0.0007268167.7968087.759304
12400.0023589470.003486207-0.0011272607.5641378.091573
12500.0022174830.004765312-0.0025478297.7186248.450522
12600.0023318040.006194275-0.0038624717.6378599.270286
12700.0021439460.007986821-0.0058428757.9445919.792876
12800.0019845310.009621407-0.0076368767.79700510.256525
12900.0021268800.010482756-0.0083558767.72557610.917056
13000.0020655460.011161493-0.0090959477.55374511.661466
14000.0014213850.011526558−0.0101051738.11109912.077521

At the finer evaluation interval, B's held-out ArmoRM gain increases progressively from 0.002325 to 0.011161 between updates 1,200 and 1,300, while A stays between 0.001985 and 0.002359. B's sequence log-ratio increases from 7.49 to 11.66 over the same interval.

The apparent jump in Figure 3 therefore reflects the 100-update evaluation interval rather than a single-step change.

Reviewer_TQ6XOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines what a reward model's scores can tell us about the policy obtained by optimizing it. It shows that even complete response rankings and reward variance need not determine whether the initial change improves target utility. Under fixed-reference Gibbs optimization, that change is governed by the covariance between the candidate reward and the target. For a linear reward class, the authors characterize the score observations needed to identify it and derive sharp intervals under a norm bound. They also give conditions under which a comparison remains valid after finite optimization, including at matched KL. Fixed response-bank experiments illustrate both successful certificates and counterexamples, while a paired RLOO run illustrates a reversal in relative performance during parameterized training.

Critical strengths

The most useful contribution is the precise distinction between a correlation observed in an experiment and a comparison that holds for every reward consistent with the available observations. Appendix A.1 makes the issue concrete: two rewards have the same ordering and variance, yet their initial effects on utility have opposite signs. The row-space condition and sharp interval then show which missing information would resolve the ambiguity.

The finite-optimization results add substance beyond this local observation. Theorem 4.3 accounts for different feasible rewards needing different multipliers to reach the same KL. The experiments also keep certified comparisons, feasible sign reversals, and unresolved cases separate. This conceptual and analytical contribution is enough for a weak accept in my view, although the practical interpretation needs the qualifications below.

Critical weaknesses

The reported small score budgets do not yet demonstrate end-to-end evaluation savings. Appendix B.4 builds the five-dimensional feature class from all five RMs' scores on the response bank, uses those candidates to fix the norm ball, and computes the target direction from target scores. These setup scores are explicitly excluded from the query count. The six- and eight-score results therefore show how much information can be discarded after this setup; they do not establish how to select a new RM cheaply without first obtaining that information.

The certificates are also sensitive to the feasible set. In Table 8, a 5% increase in the norm bound reduces the directed matched-KL certificates from 33 to 17 out of 100, and a 10% increase leaves 7. The finite experiments use exact observations within an empirical score span and a learned cardinal target. The additional target check and RLOO example help, but robustness to feature misspecification, noisy target information, and unseen policy responses remains open. I see these as limits on practical scope, rather than contradictions of the theorems.

Questions for authors
  1. In a deployment using B.4–B.5, which setup quantities could be fixed without scoring the candidate on the full response bank? How would membership of a new candidate in the feature class and norm ball be justified? This is the main clarification I need to assess the practical contribution.

  2. What supports the tight norm bound beyond its containing the observed candidates? Given Table 8, could the conclusions be stated explicitly as conditional on this calibration, or supported with a bound chosen independently of the candidates being compared?

  3. B.7 reports that four score differences plus the exact local-effect observation leave one unobserved direction in a five-dimensional class. Is one constraint redundant? Please give the ranks before and after augmentation.

2 comments

We sincerely thank Reviewer TQ6X for recognizing the conceptual and analytical contributions of our identification results and the value of the finite optimization analysis. We address each question and concern below.

W1/Q1: Setup quantities and candidate membership

We tested this directly in a fixed feature setting where the shared quantities are determined before any candidate head is trained. We use a 5 dimensional feature map from a frozen Pythia 2.8B backbone, with a fixed projection chosen before any head was trained. We then fix B=1B=1, the target direction bb, and the query locations defining AA. Five linear reward heads are subsequently trained with a Bradley–Terry loss and projected optimization. Their held out pairwise accuracies are 73.46–73.65%.

During identification, the evaluator receives only the queried scores. Full bank rewards are computed afterwards for oracle verification. We report candidate 5 against the other four heads:

Raw scores per candidateCandidate 5 comparisons certifiedOracle disagreements
427/400
640/400
840/400

Here, 40 denotes four candidate pairs under ten query selections.

1. Which setup quantities could be fixed without scoring the candidate on the full response bank?

In the theoretical setting, ϕ\phi and BB specify the reward class. Once the reference distribution and target criterion are fixed, bb is fixed as well, and AA can be selected before candidate scores are observed. Thus, ϕ\phi, BB, bb, and AA can all be fixed before evaluating a candidate; only e=Awe=Aw is candidate specific. The supplementary experiment follows this order explicitly.

2. How would membership of a new candidate in the feature class and norm ball be justified?

A candidate family can use a fixed shared reward representation, with each candidate represented by its own linear reward head: rw(x,y)=w⊤ϕ(x,y)r_w(x,y)=w^\top\phi(x,y).

In this setting, feature class membership is determined by the candidate parameterization rather than inferred from full bank scores. The norm constraint can likewise be imposed during candidate training. In our experiment, ww is projected onto the B=1B=1 ball specified before training after every update, so both conditions hold before evaluation without requiring full bank candidate scoring.

W2/Q2: Norm calibration and robustness

1. What supports the tight norm bound beyond its containing the observed candidates?

In the pretrained RM analysis, BB is an empirical calibration of the feasible reward class, and the reported certificates are therefore conditional on this calibration. We now state this explicitly.

2. Given Table 8, could the conclusions be stated explicitly as conditional on this calibration, or supported with a bound chosen independently of the candidates being compared?

We have revised the caption of Table 3 to state explicitly that “Certificate counts are conditional on the empirical class and norm-bound calibration in Appendix B.4.”

We also tested a bound chosen independently of the candidates being compared. In the fixed-feature experiment above, B=1B=1 is specified before any candidate head is trained. With this independently fixed bound, the newly introduced candidate is certified in 40/40 local comparisons at both the 6-score and 8-score budgets.

Target score perturbations. Keeping the query locations fixed, we recomputed the matched KL 0.20.2 certificates under 20 random target perturbations at each level:

Target perturbation RMSDirected finite certificates
0%33.00/100
10%32.65/100
25%32.40/100
50%31.95/100

Independently sampled responses. For the 33 directed finite comparisons certified on the primary Qwen bank at matched KL 0.20.2, all 33/33 retain the same candidate ordering on a second Qwen response bank sampled with the same prompts and generation settings.

Q3: Ranks in Appendix B.7

The four queried raw scores come from the same prompt, so they induce only three independent within-prompt score differences. Accordingly, A∈R3×5A\in\mathbb{R}^{3\times 5} and rank⁡(A)=3\operatorname{rank}(A)=3.

Appending the normalized exact local-effect row gives A+∈R4×5A_{+}\in\mathbb{R}^{4\times 5} and rank⁡(A+)=4\operatorname{rank}(A_{+})=4, so dim⁡ker⁡(A+)=1\dim\ker(A_{+})=1. SVD and pivoted QR independently confirm ranks 3 and 4.

The feasible assignments in Appendix B.7 and Figure 4 vary along this remaining direction while preserving the three observed score differences and each candidate's exact local effect, and they attain opposite finite-comparison signs. We revised Appendix B.7 and the Figure 4 caption to state the raw score/difference distinction and the ranks directly.

Thank you for the additional experiments and clarifications. The fixed-feature experiment addresses my main concern, and the rank clarification resolves my third question. I maintain my support for the acceptance recommendation.

Reviewer_wosQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether static evaluation of reward models (RMs) can help determine which RM will lead to better downstream policy performance under KL-regularized optimization. The main observation is that the covariance between the candidate RM and the target reward is more informative about initial policy improvement than ranking accuracy or reward variance alone. The paper then derives conditions under which evaluating only a subset of RM scores is sufficient to determine which candidate RM is better, and studies how this comparison changes as the policy moves further from the reference policy in terms of KL distance. The experiments use scores from multiple pretrained RMs, with ArmoRM as the target reward, to validate the theoretical results. An additional experiment performs actual policy training using RLOO with two candidate RMs, showing how their relative downstream performance can change over training.

Critical strengths

Problem Motivation. The paper addresses an important question in reward model evaluation. Selecting an RM based on static evaluation before policy training is common practice, and understanding how well these evaluations predict downstream policy improvement is useful. I particularly like that the paper considers both the initial policy improvement and how the comparison between RMs changes as optimization progresses. Additionally, it would be valuable to know whether evaluating only a subset of responses can help identify the better candidate RM, given the cost of training a policy for every candidate RM.

Critical weaknesses

Writing and Presentation. I find the paper quite difficult to follow, even after careful reading. The presentation is very notation-heavy, with several definitions and theoretical results introduced without sufficient intuition or motivation. For example, Section 3.3 introduces the feature function ϕ\phi, observation matrix AA, and norm bounds without clearly explaining how these quantities are constructed in practice, which is only clarified in Appendix B.4. Similarly, Theorem 4.3 introduces several additional quantities whose definitions and interpretations are difficult to follow without going through the appendix. I also find that the simple example in Appendix A.1, which illustrates why ranking and variance are insufficient, should have been included in the main text, as it provides important intuition for the central idea of the paper. Overall, I think the paper would benefit significantly from a simpler presentation, clearer definitions, and more intuition behind the theoretical results.

Novelty, Practical Applicability, and Experiments. While the problem is interesting, I am not sure how much of the underlying idea is new. The relationship between reward covariance and policy improvement is already well studied, and related work such as Correlated Proxies (Laidlaw et al., ICLR 2025) does not seem to be discussed. The main contribution appears to be the partial-identification results, but I have some concerns about their practical applicability. In particular, the method requires information about the target reward, and the experiments construct the linear reward class using the complete scores of all candidate RMs. If these scores are already available, it is unclear what practical benefit is gained from evaluating only a small subset of responses. Additionally, I found the experiments quite difficult to understand in terms of what exactly they demonstrate. Most experiments use exact Gibbs optimization over fixed response banks, while actual policy training is evaluated with only one pair of RMs. I am not sure what conclusions we can draw about the usefulness of the proposed approach for practical RM selection, or how well the theoretical results translate to actual RL training.

References

Laidlaw, C., Singhal, S., & Dragan, A. (2025, May). Correlated proxies: A new definition and improved mitigation for reward hacking. In International Conference on Learning Representations (Vol. 2025, pp. 88379-88421).

Questions for authors
  1. Evaluation Cost: How many RM and target-reward evaluations does the proposed approach require in total?

  2. Access to Target Reward: The proposed framework requires access to the target reward rG. How would this work in practice, given that the true target reward is generally unknown in RLHF? Is the assumption that we have access to another reliable RM, as in the experiments with ArmoRM?

  3. Relation to Existing Work: How does the proposed framework differ from Correlated Proxies (Laidlaw et al., ICLR 2025)? The relationship between proxy-target reward correlation and downstream policy improvement appears closely related to the ideas explored in this paper.

4 comments

We sincerely thank Reviewer wosQ for recognizing the strong motivation behind the RM selection problem and the value of studying both initial and finite optimization behavior. We address each question and concern below.

W1–W2: Writing and presentation

Thank you for the feedback on presentation. Additionally, we have revised the manuscript accordingly and uploaded a new version. The main text changes are summarized below.

  1. Section 3.3. Before Theorem 3.2, we now explain the identification problem directly: within the specified linear reward class, the local effect is identified exactly when every reward variation left unconstrained by the observations has zero effect on the initial change in target utility. We also explain how the notation is instantiated in the experiments. The span of the five candidates' prompt centered score vectors defines the shared feature coordinates Φ\Phi; selected within prompt feature differences form the rows of AA; centered target scores define bb; and the shared norm bound together with the observed candidate scores defines each feasible set.

  2. Section 4.4 / Theorem 4.3. We added the intuition for the finite matched KL comparison before stating the theorem. Different feasible rewards may require different optimization strengths to reach the same KL distance, so evaluating all rewards at one common optimization strength does not answer the matched KL comparison. We now explain why the theorem uses a finite cover of each feasible set, the reachability condition, and the bounds that transfer the computed gains to the full feasible set.

  3. Section 3.2. We moved the three response example from Appendix A.1 into the main text. The two candidate rewards have the same strict ordering and the same reward variance, but their covariances with the target have opposite signs, giving opposite initial effects on target utility.

These revisions clarify the presentation; the assumptions and theoretical results are unchanged.

W3/Q3: Relation to Correlated Proxies

Thank you for pointing out the connection to Correlated Proxies[1]. We have also revised the Related Work section and Section 4.4 to make this connection explicit. We discuss the relation between our matched KL quantity and reference policy proxy target alignment, as well as the difference in the decision problem being studied.

The two works focus on different stages of the RLHF pipeline:

Correlated Proxies studies the stage after a proxy reward has been specified, analyzing how the proxy and true reward relate under the reference policy and how reference policy regularization can control reward hacking during subsequent policy optimization. Our paper studies the earlier reward model evaluation and selection stage: before policy optimization, when does static evidence about candidate reward models determine which candidate leads to higher downstream target utility?

We first show that even complete response ordering together with equal reward variance does not determine the sign of the initial target-utility effect: two rewards satisfying both can have opposite covariances with the target and therefore opposite initial effects. Our results give an identification theory for this selection problem. For a specified linear reward class, Theorem 3.2 gives a necessary and sufficient condition for identifying the local downstream effect from static observations. When point identification fails, Theorem 3.3 gives the sharp identified interval, and Corollary 3.4 shows that the pairwise reward model comparison can be identified even when neither individual effect is point identified. We then extend the analysis beyond the reference policy: Theorem 4.1 and Corollary 4.2 account for policy movement, while Theorem 4.3 gives a uniform comparison certificate over the entire feasible reward sets at a specified finite KL distance. At matched KL, Theorem A.4 gives the scale normalized local quantity MKL=M0/RV0M_{\mathrm{KL}} = M_0/\sqrt{RV_0}, which has the reference policy correlation interpretation discussed in Section 4.4.

Thus, Correlated Proxies asks how to optimize a specified proxy, whereas our paper asks what static reward model evidence is sufficient to select among candidate reward models before policy optimization.

W4: Practical benefit of partial evaluation

The identification results determine when limited static evidence is sufficient for RM selection. Theorems 3.2–3.3 characterize whether the local effect M0M_0 is identified and, when it is not, give a sharp uncertainty width controlled by RΓidR\Gamma_{\mathrm{id}}. Corollary 3.4 gives a sharp interval for the pairwise comparison, which can determine the winner even when neither individual effect is point identified. Theorem 4.3 extends this comparison to finite matched KL optimization.

In Appendix B.4, the complete score panels are used to instantiate the empirical reward class. To directly test the setting where this shared setup is fixed before evaluating a new candidate, we added a fixed-feature experiment in which ϕ\phi, BB, bb, and AA are fixed before any candidate head is trained, and the evaluator receives only the queried candidate scores.

Raw scores per candidateCandidate 5 comparisons certifiedOracle disagreements
427/400
640/400
840/400

Here, 40 denotes four comparisons involving the new candidate under ten query selections. Full-bank scores are computed only afterwards for oracle verification. Thus, once the shared setup is fixed, a new candidate can be evaluated from sparse candidate-specific scores without first scoring the full response bank.

W5: What the experiments are designed to test

The experiments serve different purposes, corresponding to the three questions in the paper.

  1. Why fixed response banks and exact Gibbs optimization?
    The theory studies fixed reference KL regularized reward optimization, for which the Gibbs policy is the exact distribution space optimizer. Fixed response banks keep the support, candidate scores, target scores, and feasible reward sets known. The complete score panels are used to instantiate the empirical linear reward class and feasible sets on real pretrained RM scores, so the identification claims can be checked directly without additional effects from parameterized policy training.

  2. What do the fixed bank experiments establish?
    For RQ1, the controlled construction keeps complete ordering and reward variance fixed while obtaining opposite signs of M0M_0 and opposite downstream gains, showing that these static quantities do not determine the downstream direction.

    For RQ2, the partial observation experiments show both identified and nonidentified comparisons, corresponding to Theorems 3.2–3.3 and Corollary 3.4.

    For RQ3, the matched KL experiments evaluate whole feasible reward sets after finite optimization: some comparisons are certified by Theorem 4.3, while in others feasible rewards attain opposite finite orderings.

  3. What is the role of the RLOO experiment?
    The RLOO experiment asks whether the horizon dependence in RQ3 can also occur under parameterized policy training. Figure 3 gives such an example: two RMs with similar static accuracy change their relative target utility over training.

Q1: Evaluation cost

For the primary Qwen response bank, we score 4,096 responses with five candidate RMs and ArmoRM. This gives 20,480 candidate RM scores and 4,096 target scores, or 24,576 model–response scores in total. Across the four response banks used in the static analyses, the total is 73,728 model–response scores.

These scores are computed once and reused in the later identification and certification analyses, so those calculations do not require additional RM scoring. We have added this setup cost accounting explicitly to Appendix B.4 in the revised manuscript.

Q2: Access to Target Reward

Yes. In our experiments, ArmoRM is the specified target evaluator. The theory does not require the target to be another pretrained RM. It requires a downstream criterion rGr_G, because whether one candidate RM is better than another is defined with respect to the resulting target utility.

For the experiments, a specified target is needed to evaluate the quantities studied by the theory, including the local target utility effect, the pairwise RM comparison, and the finite target utility gain. This type of controlled setup is also used in closely related theory driven RM work; for example, Razin et al.[2] use ArmoRM as the ground truth reward when studying downstream reward optimization.

This experimental construction does not require the theory to assume that a unique true target is known in practice. If the target is only known to belong to a set G\mathcal G, Theorem D.1 extends the identification condition to that set: span⁡{b(g):g∈G}⊆row⁡(A)\operatorname{span}\{b(g):g\in\mathcal G\}\subseteq\operatorname{row}(A).

This gives simultaneous local identification for every target in the specified set rather than only for one fixed rGr_G.

If no particular target is specified and identification is required for every bounded target reward, Corollaries D.2–D.3 show that the only reward variation that may remain unobserved is h(x,y)=c(x)h(x,y)=c(x), a prompt wise additive constant, which leaves the entire Gibbs policy path unchanged.

[1] Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking. ICLR 2025.

[2] Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What Makes a Reward Model a Good Teacher? An Optimization Perspective. NeurIPS 2025.

Thank you to the authors for their response and revisions. I appreciate the improvements and clarification regarding Correlated Proxies as well as the additional fixed-feature experiment. However, I remain unconvinced about the practical applicability of the proposed method for RM selection in policy training. While the experiments show that a small number of candidate RM scores can identify comparisons under a fixed feature space, it remains unclear to me how the results translate to actual policy training. In particular, the RLOO experiment demonstrates that the relative performance of two RMs can change during training. However, it does not show whether the proposed identification method could reliably select the better RM before training, at a specified training horizon or policy KL distance. Given that the motivation of the paper is to select a better RM without training a policy for every candidate, I think this is an important missing connection between theory and application. Overall, some of my concerns have been addressed, but I remain unsure about the points I have raised above, and hence I will maintain my current score, albeit my confidence remains low, as I might have missed some important details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.