- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how static reward model evaluations are connected to downstream policy optimization. Interestingly, they show that the initial improvement of the policy depends on the covariance of the proxy reward model and the target reward model. They then show that the responses that the reward model is measured on can matter most for uncovering this initial reward model optimization effect. In addition, they study how finite policy optimization affects reward model comparison. Most of this work is theory driven, but they also provide a variety of real reward modeling experiments.
Critical strengths
- RQ1 is a novel and useful finding.
- Although RQ2 and RQ3 are not that interesting to me, they do provide a detailed extension and further study of the ideas of the paper, making it a complete study.
- This paper provides both theoretical results and empirical analysis.
Critical weaknesses
- To me the story of the paper is not that clear (although the punchlines are). Although I am very excited about the results, the paper is still hard to follow.
- RQ1 seems to be the main contribution of the paper. It is an interesting finding. Otherwise, I do not find RQ2 or RQ3 to be too informative.
Questions for authors
- Can you explain the setting of Figure 2a? Is this online policy optimization assuming the same responses are sampled from the policy for each RM?
- Why does the policy score suddenly jump late in training in Figure 3? This is very strange.
2 comments
Q1: Setting of Figure 2a
Figure 2a is not online policy optimization. It uses one fixed empirical response bank with 32 responses for each of 128 prompts, generated once and reused for rewards A–C.
For each prompt, Best-of- samples response indices uniformly with replacement from this bank and selects the highest-ranked sampled response. Rewards A–C have the same complete ordering and use the same fixed tie-breaking rule, so they select the same response from every common candidate set. The plotted curve is the exact expected target gain on this bank, computed from the rank-selection probabilities in Appendix B.2.
Figure 2a therefore isolates the effect of ranking: identical orderings give identical Best of outcomes. Figure 2b uses exact Gibbs optimization on the same bank, where reward magnitudes affect response probabilities and lead to different target gains at matched KL.
Q2: Why does the policy score suddenly jump late in training in Figure 3?
Figure 3 evaluates held-out utility every 100 updates, so the behavior between updates 1,200 and 1,300 is not shown.
-
We checked the training and evaluation setup over this interval. The held-out prompts, sampling settings, seed rule, learning rate, and KL coefficient were unchanged.
-
We resumed from the update-1,200 checkpoint and evaluated every 10 updates through 1,300.
| Update | A gain | B gain | A sequence log-ratio | B sequence log-ratio | |
|---|---|---|---|---|---|
| 1100 | 0.002278621 | 0.003071451 | −0.000792829 | 6.843555 | 5.084101 |
| 1200 | 0.002197895 | 0.002325078 | -0.000127183 | 7.443717 | 7.494322 |
| 1210 | 0.002286340 | 0.002518624 | -0.000232284 | 7.479709 | 7.593199 |
| 1220 | 0.002091772 | 0.002411953 | -0.000320181 | 7.683397 | 7.611277 |
| 1230 | 0.002174665 | 0.002901481 | -0.000726816 | 7.796808 | 7.759304 |
| 1240 | 0.002358947 | 0.003486207 | -0.001127260 | 7.564137 | 8.091573 |
| 1250 | 0.002217483 | 0.004765312 | -0.002547829 | 7.718624 | 8.450522 |
| 1260 | 0.002331804 | 0.006194275 | -0.003862471 | 7.637859 | 9.270286 |
| 1270 | 0.002143946 | 0.007986821 | -0.005842875 | 7.944591 | 9.792876 |
| 1280 | 0.001984531 | 0.009621407 | -0.007636876 | 7.797005 | 10.256525 |
| 1290 | 0.002126880 | 0.010482756 | -0.008355876 | 7.725576 | 10.917056 |
| 1300 | 0.002065546 | 0.011161493 | -0.009095947 | 7.553745 | 11.661466 |
| 1400 | 0.001421385 | 0.011526558 | −0.010105173 | 8.111099 | 12.077521 |
At the finer evaluation interval, B's held-out ArmoRM gain increases progressively from 0.002325 to 0.011161 between updates 1,200 and 1,300, while A stays between 0.001985 and 0.002359. B's sequence log-ratio increases from 7.49 to 11.66 over the same interval.
The apparent jump in Figure 3 therefore reflects the 100-update evaluation interval rather than a single-step change.
We sincerely thank Reviewer 2wjo for recognizing RQ1 as a novel and useful finding, and for valuing the paper’s combination of theoretical results and empirical analysis. We address each question and concern below.
W1: Story and presentation
RQ1 establishes the starting point: even complete response ordering together with equal reward variance can leave the sign of the initial target utility change undetermined.
Additionally, we have revised the manuscript accordingly and uploaded a new version. In Section 3.2, we moved the three response example into the main text. In Section 3.3, before Theorem 3.2, we now explain what information must be identified and how the notation maps to the empirical construction. In Section 4.4, before Theorem 4.3, we added the intuition for why matched KL requires comparison over the full feasible reward sets rather than at one common optimization strength.
These are presentation changes; the assumptions and theoretical results are unchanged.
W2: Why RQ2 and RQ3 are needed
RQ1 shows that ordering and reward variance can be insufficient, but it leaves open the question of what static information is sufficient. RQ2 answers this. For the specified linear reward class, Theorem 3.2 gives the necessary and sufficient identification condition, Theorem 3.3 gives the sharp identified interval when point identification fails, and Corollary 3.4 shows that the pairwise RM comparison can be identified even when neither individual local effect is point identified.
A second issue remains after the local comparison is identified: policy optimization changes the response distribution, so the local ordering need not persist. RQ3 studies this step. It separates the uncertainty already present in the static observations from the additional effect of policy movement, and gives finite comparison guarantees over the feasible reward sets, including at matched KL. The reversal constructions show why the local result alone is not enough for the finite optimization question.
So the role of RQ2 and RQ3 is to answer the two questions left open by RQ1: what static information resolves this ambiguity, and whether the resulting conclusion still holds after the policy moves.