acceptodds
Withdrawn from ICLR 2027

Beyond Unbiasedness: Conservative Reward Modeling from Observational Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reward models (RMs) are central to reinforcement learning from human feedback (RLHF) for aligning large language models, while observational user feedback provides a scalable and deployment-aligned source of supervision. However, such feedback is selectively observed, inducing selection bias between the observed data and the target population. Existing propensity-based methods can correct this bias, but we show that unbiasedness alone does not guarantee reliable reward modeling. In particular, long-tailed observational feedback leaves low-support regions with near-zero propensities, where inverse weighting can amplify sparsely supported observations into spurious reward peaks that attract downstream policy optimization. To address this problem, we propose Conservative Reward Modeling (ConRM), which combines clipped propensity weighting with support-aware conservative regularization. Clipping limits the influence of near-zero-propensity samples, while the conservative regularizer uses well-supported anchors and local reward variations to construct data-supported intervals that constrain low-support extrapolation. Experiments on three public preference datasets under simulated long-tailed observational feedback demonstrate that ConRM consistently improves reward estimation across datasets and model backbones, while also yielding more reliable downstream policy alignment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, reward model, RLHF, alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ygEaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to train a reward model given a dataset with selectively observed user feedback. This paper designs a heuristic method that combines propensity clipping with a conservative regularization to constrain reward extrapolation in low-support regions. Experiments in this paper demonstrate its advantage in reward estimation and downstream RLHF safety performance on three datasets that do not include selection bias, but on which selection bias was simulated.

Critical strengths

The paper studies an interesting problem in reward modeling: selection bias may exist in principle and may affect the generalizability of the learned reward models.

The proposed method is intuitive and can be integrated with IPS and DR estimators.

Critical weaknesses

Lack of soundness. The computation of reward intervals seems self-referential. The method uses the model's own reward landscape to determine whether its predictions in another region are reliable. This seems to be problematic. In addition, does being "well-supported" imply more reliable reward estimation? This question also lacks justification.

Insufficient Evidence. The paper’s motivation appears to follow: Sparse feedback → Spurious reward peaks → Policy exploitation. The experiments only show the performance, but not follow the motivation at all. In addition, assumption 1 seems overly strong, and there are no empirical support to its practical feasibility. By the way, the paper does not provide sensitivity analyses on the two key hyperparameters

Questions for authors

The paper aims to address long-tailed observational user feedback, but it only evaluates on simulated datasets instead of real-world observational feedback. This is like developing a car and testing it by simulation, without ever testing its safety under real driving conditions. Would consumers trust such a car?

It seems the paper lacks implementation and experimental details to ensure reproducibility and does not seem to release the related code. Will the related code be released? Are the results in this paper reproducible?

Reviewer_RVxAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Conservative Reward Modeling (ConRM) that combines clipped inverse propensity weighting with support-aware conservative regularization. Experiments on three preference datasets show some results.

Critical strengths

This paper studies a meaningful and practical question about preference debiasing and reward hacking.

Critical weaknesses
  1. Poor writing. Are all 24 equations in your main text necessary? Please carefully polish your manuscript to improve its clarity and readability.

  2. The most significant weakness of this paper lies in its insufficient experimental evaluation: (1) Lack of direct evaluation of RM quality. Only RMSE, R2R^2, and MSE on HelpSteer, SafeRLHF, and UltraFeedback are reported. These results only demonstrate the models' fitting performance, rather than providing convincing evidence of effective debiasing. This concern is further amplified by the fact that these datasets are relatively dated and inherently noisy. (2) Lack of comprehensive evaluation in RLHF settings. Reward modeling is one of the most critical components of RLHF, yet the paper provides insufficient evidence of the proposed method's effectiveness in actual RLHF pipelines. Table 3 offers very limited insights: it remains unclear whether the observed improvements stem from the underlying RM itself or from the RM augmented with ConRM. Moreover, the paper fails to establish whether ConRM offers meaningful advantages over existing debiasing approaches for reward modeling. Even comparisons with classical baselines, such as the Bradley–Terry RM, are missing.

Questions for authors

Please refer to the weakness.

Reviewer_HF1hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reward modeling from selectively observed user feedback. Its central argument is that statistical unbiasedness does not necessarily make a reward model a reliable target for downstream policy optimization: observations with very small propensities can receive excessive inverse-propensity weights, amplifying estimation errors.

The proposed Conservative Reward Modeling framework, ConRM, combines propensity clipping with a regularizer for low-support regions. For each low-support example, it selects a nearby high-support anchor and uses directional derivatives of the reward model around that anchor to construct lower and upper reward bounds. Predictions outside those bounds are penalized.

The paper presents variance expressions for IPS and DR estimators and a conditional interval-coverage proposition based on an extrapolation assumption. Experiments evaluate reward prediction on three datasets with simulated selective observation and downstream RLHF performance on four safety benchmarks.

Critical strengths
  1. The problem is practically important and goes beyond average prediction accuracy. A reward model is actively optimized by the policy, so a low average prediction error does not rule out exploitable local overestimation. The paper identifies a worthwhile problem and attempts to address both excessive sample influence and unsupported extrapolation.
  2. The paper provides preliminary evidence across datasets and includes component ablations. Table 1 compares several debiasing and stabilization methods, while Figure 2 separately examines clipping and regularization. On HelpSteer, ConRM-DR achieves an RMSE of 0.3933 compared with 0.4120 for MRDR. These results support further investigation, although they do not yet establish the proposed mechanism.
Critical weaknesses

1.Under the current formulation, the DR reward-model update may reduce to the IPS update, leaving a gap between the method description and the reported results.

The imputation term is defined as a separately parameterized predictor of the pointwise loss. Algorithm 1 updates that predictor before updating the reward model. If the imputation model and propensity estimates are held fixed during the reward-model update, the imputation term contributes no gradient to the reward-model parameters. The resulting gradient of the clipped DR objective is then identical to that of clipped IPS.

Under this interpretation, the imputation term does not provide a distinct reward-learning signal. If the implementation instead relies on shared parameters, a differentiable loss constructed from imputed labels, or another coupling mechanism, this must be explicitly specified. Otherwise, the learning mechanism of the DR branch and its empirical differences from IPS cannot be adequately understood from the manuscript.

2.The central claim that the intervals constrain spurious reward peaks lacks sufficiently independent theoretical and empirical support.

Both the anchor values and the directional derivatives used to construct the bounds come from the reward model being trained. A high observation propensity does not itself establish that an anchor’s reward prediction is accurate. Assumption 1 effectively assumes that behavior in the low-support region is controlled by neighboring variations, while Proposition 1 leaves the estimation-error allowance without a computable or finite-sample bound. The result therefore does not establish that the intervals actually used during training cover the true reward.

A concrete boundary case illustrates the limitation: if the relevant examples lie in a common affine region of the reward function, the constructed lower and upper bounds both equal the model’s own prediction at the target example. The regularization penalty is then zero, even if the model is systematically overestimating rewards there. This does not show that the method is always ineffective, but it demonstrates that the construction does not automatically constrain unreliable extrapolation.

Moreover, the downstream experiments in Table 3 compare only against Naive, omitting IPS, DR, and clipping-only alternatives. They also do not directly measure policy exploitation of reward errors. The reported improvements therefore cannot be attributed specifically to the proposed interval constraint or to reduced reward hacking.

Questions for authors

1.Implementation clarification: Please provide the complete computational graph for the DR branch. Does the imputed loss depend on the reward-model parameters during their update? If so, how should the current objective be revised to express that dependence? If not, what produces a learning signal distinct from IPS?

2.Theoretical and implementation clarification: Are gradients stopped through the interval boundaries? Can you report empirical coverage, interval width, and reward overestimation in low-support regions? How does the method address the affine-region case in which its regularizer is identically zero?

3.Critical supporting evidence: Under matched policy-training settings, does ConRM improve downstream performance over IPS, DR, and clipping-only versions? Can you also report independent quality assessments or utility on benign requests to distinguish improved reward reliability from a stronger tendency to refuse?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.