acceptodds
Withdrawn from ICLR 2027

Don’t Relearn What You Already Know: using Residual Reward Learning for RLHF

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human preferences are expensive, yet experts can often specify some of the components of a desired reward programmatically. Recent work has shown that such proxy rewards can be combined with preference learning by learning only their residual. But why does residual reward learning outperform simply adding a proxy to a conventionally learned reward model, and when should we expect it to do so? We identify two mechanisms. First, residual learning preserves the decomposition of the true reward, whereas naive addition double-counts components already captured by the proxy. Second, when the proxy explains sufficient variation in the true reward, the residual constitutes a lower-variance and therefore simpler learning target. We theoretically characterize these conditions and test them through controlled experiments across eight RL environments. Finally, we demonstrate that the same principle extends to LLM alignment by combining a programmatic proxy with human preferences for summarization. Our results characterize when prior reward knowledge can reduce the burden placed on preference learning.

Primary area:
reinforcement learning
Keywords:
RLHF, Residual Learning, RL, Reward Modeling, Reward Design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5GBAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to use a known proxy reward in preference-based reward learning. Building on residual reward learning, it compares adding the proxy after training a reward model with including the proxy in the reward model’s training objective. The authors analyze reward double-counting and target variance reduction, and evaluate the approaches across 8 RL environments and GPT-2 summarization with PPO.

Critical strengths
  1. The paper clearly explains why the proxy should be considered during reward learning.

    Adding a proxy after training and including it during training can produce different results, even with the same proxy and preference data. The paper explains this distinction through the training objectives and reward decomposition. It acknowledges the existing RRM formulation and focuses on understanding why and when it helps.

  2. A controlled comparison of policy performance

    The GPT-2 experiments use the same preference data, initial SFT policy, and PPO settings. Generated summaries are evaluated by a separately trained judge that is not used during PPO. This helps assess whether the difference in reward-model training leads to better policy performance.

Critical weaknesses
  1. The theoretical explanation for improved learning efficiency needs additional assumptions.

    Appendix A.4 argues that lower target variance leads to lower estimator variance and fewer required samples. However, this conclusion does not follow from the stated independent-noise assumption alone. For example, with fixed inputs and the same observation noise, subtracting a known linear proxy can reduce target variance while leaving the variance of an ordinary least-squares estimator unchanged. I think the main paper and appendix don’t fully resolve this gap.

  2. Table 1 needs a minor formatting correction.

    The overall style of Table 1 differs from the sample table in the ICLR template. Its caption is also placed below the table.

  3. The experiments do not fully test the claimed learning efficiency and conditions for improvement.

    The comparison across eight environments provides useful evidence of improved policy performance. However, the paper does not vary the preference budget to show how much feedback residual learning saves, or systematically vary proxy quality within the same environment to test when it helps. The LLM experiment also lacks vanilla reward-only and proxy-only PPO baselines.

Questions for authors
  1. Which estimator and additional assumptions support the relationship between target variance and estimator variance in Appendix A.4? How does this account for cases where subtracting a known linear proxy leaves the least-squares estimator variance unchanged? How does the analysis extend to Bradley–Terry preference learning?
  2. Figure 4 computes MSE after affine-aligning predictions to each target. Since this MSE also depends directly on target variance, the reported correlation may partly reflect that dependence. Does residual learning still show an advantage when MSE is normalized by target variance, or when both methods are evaluated on the same held-out preference pairs?
  3. In the LLM experiment, how does residual learning compare with PPO using only the vanilla reward model or only the proxy reward? These baselines would help show whether incorporating the proxy improves over standard preference-based reward learning, beyond outperforming naive reward addition.
Reviewer_bUguOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a Bradley-Terry (BT) reward model training scheme to integrate programmatic (proxy) rewards with human preference labels. They show that integrating them by learning a “residual” reward improves the performance of the reward model. The method is tested on a range of RL environment and on LLM finetuning from human feedback.

Critical strengths

The motivation and derivation of the method are very clear. The method is flexible and easy to implement; it seems robust to adversarial rewards in the proxy and can be applied to a wide range of different tasks and setups (from language to video games).

Critical weaknesses

The analysis could have been pushed further. I would have liked to see experiments about label efficiency: for the same performance, how many fewer labels does the residual method need? Can we target labeling efforts to get the best performance at the lowest budget (maybe the proxy can help select the most informative samples to label)?

The quality of the judge for the LLM experiment (accuracy of 71%) is low; a better judge may be needed for it to be considered “ground truth”.

Questions for authors

Do you have experiments showing the performance at lower labeling budgets or experiments to show how to target the labeling? This is different from the QBC experiment in appendix and would be more about how to leverage the proxy.

Could you try a better judge for the evaluation of the LLM experiments? There are a few open-weight preference reward models available and they could be a stronger judge.

In the 8 RL environments, do the vanilla and partial methods get the adversarial reward scheme as well? Vanilla sometimes collapses, but not in the environment where the adversarial reward is enabled.

Reviewer_UVPaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds on the residual reward learning formulation introduced by Cao et al. (2026), and further investigates the question of why and when conditioning preference-based reward learning on a known proxy reward outperforms naive proxy integration, which adds the proxy to an independently learned reward model has been trained. It examines two explanations for this benefit: avoiding double-counting of information already captured by the proxy, and potentially reducing the variance of the remaining learning target. To test the proposed theoretical explanation, the paper designs controlled experiments across eight RL environments with access to ground-truth rewards, and an extension to LLM alignment that combines a programmatic summary-quality proxy with human preferences for GPT-2 summarization. The results show that residual composition attains the highest ground-truth return in all eight environments, and that in the summarization task fine-tuning with residual reward learning also achieves the improvement over naive composition.

Critical strengths

S1. The paper turns an overlooked methodological question into a clean controlled test, and the result is consistent.

Combining a programmatic proxy reward with preference learning is already common practice (RBR, HERO, RLMR). All existing work adds the proxy after training. Prior work has not tested whether the stage at which a known proxy enters the reward pipeline matters. This paper poses that question and examines it directly. The naive and residual conditions share the same architecture, the same preference data, the same proxy, and the same hyper-parameters. The only difference is whether the proxy enters the Bradley–Terry likelihood. Across eight environments with ground-truth rewards and 10 seeds, residual learning achieves the highest return in all eight.

Critical weaknesses

W1. The naive baseline is fixed to unit-weight addition. The paper's claimed mechanism, double-counting, is therefore confounded with scale calibration.

The double-counting argument:

fnaive+=f∼+f∗=2f∼+fΔf^{+}_{\text{naive}} = f^{\sim} + f^{*} = 2 f^{\sim} + f^{\Delta}

rests on a hidden assumption: adding the proxy at weight 11 to an independently learned reward model. But the actual harm of double-counting depends entirely on the relative scale of f∼f^{\sim} and f^\hat{f}, and the paper normalizes the proxy to match the reward model's output range by hand. A naive baseline with a tuned scalar weight is not be tested.

Prior methods such as RLMR use weighting or normalization, so unit-weight addition is not the only standard naive baseline.

Questions for authors

Q1. How much of the residual advantage against a weighted naive proxy integration baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.