- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how to use a known proxy reward in preference-based reward learning. Building on residual reward learning, it compares adding the proxy after training a reward model with including the proxy in the reward model’s training objective. The authors analyze reward double-counting and target variance reduction, and evaluate the approaches across 8 RL environments and GPT-2 summarization with PPO.
Critical strengths
-
The paper clearly explains why the proxy should be considered during reward learning.
Adding a proxy after training and including it during training can produce different results, even with the same proxy and preference data. The paper explains this distinction through the training objectives and reward decomposition. It acknowledges the existing RRM formulation and focuses on understanding why and when it helps.
-
A controlled comparison of policy performance
The GPT-2 experiments use the same preference data, initial SFT policy, and PPO settings. Generated summaries are evaluated by a separately trained judge that is not used during PPO. This helps assess whether the difference in reward-model training leads to better policy performance.
Critical weaknesses
-
The theoretical explanation for improved learning efficiency needs additional assumptions.
Appendix A.4 argues that lower target variance leads to lower estimator variance and fewer required samples. However, this conclusion does not follow from the stated independent-noise assumption alone. For example, with fixed inputs and the same observation noise, subtracting a known linear proxy can reduce target variance while leaving the variance of an ordinary least-squares estimator unchanged. I think the main paper and appendix don’t fully resolve this gap.
-
Table 1 needs a minor formatting correction.
The overall style of Table 1 differs from the sample table in the ICLR template. Its caption is also placed below the table.
-
The experiments do not fully test the claimed learning efficiency and conditions for improvement.
The comparison across eight environments provides useful evidence of improved policy performance. However, the paper does not vary the preference budget to show how much feedback residual learning saves, or systematically vary proxy quality within the same environment to test when it helps. The LLM experiment also lacks vanilla reward-only and proxy-only PPO baselines.
Questions for authors
- Which estimator and additional assumptions support the relationship between target variance and estimator variance in Appendix A.4? How does this account for cases where subtracting a known linear proxy leaves the least-squares estimator variance unchanged? How does the analysis extend to Bradley–Terry preference learning?
- Figure 4 computes MSE after affine-aligning predictions to each target. Since this MSE also depends directly on target variance, the reported correlation may partly reflect that dependence. Does residual learning still show an advantage when MSE is normalized by target variance, or when both methods are evaluated on the same held-out preference pairs?
- In the LLM experiment, how does residual learning compare with PPO using only the vanilla reward model or only the proxy reward? These baselines would help show whether incorporating the proxy improves over standard preference-based reward learning, beyond outperforming naive reward addition.