acceptodds
Under review as a conference paper at ICLR 2027

Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF

Abstract

Feedback noise is pervasive in preference data, arising from subjective human judgments, annotation mistakes, and imperfect LLM judges. Standard reward modeling trained on such feedback yields a biased estimate of the clean risk. Although classical denoising methods are directly applicable to noisy feedback learning, they generally assume a uniform noise ratio across samples. However, this assumption fails for feedback data, where noise is intrinsically sample-wise. To address this challenge, we propose CleanRM, which uses a few clean feedback to calibrate sample-wise noise ratios via optimal transport. Specifically, CleanRM aligns the joint distribution of preferences reverted from noisy feedback with that of clean feedback and then trains the reward model using a noise-corrected surrogate loss. Theoretically, we prove that the resulting objective is unbiased with respect to the clean risk. Extensive experiments demonstrate that CleanRM outperforms state-of-the-art denoising baselines and yields more effective downstream RLHF alignment. Code is available at https://anonymous.4open.science/r/CAN-RM-46BB.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model, Reward Modeling, Noisy Feedback, Optimal Transport, Clean Anchor

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_frJmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CleanRM, a framework for reward modeling under sample-wise feedback noise. It uses a few clean feedback to calibrate sample-wise noise ratios. It constructs a noise-corrected surrogate loss that is theoretically unbiased with the clean risk. Experiments demonstrate improved reward modeling performance over existing baselines, and downstream GRPO experiments show consistent improvements in policy safety.

Critical strengths
  1. The paper explicitly deals with sample-wise heterogeneous feedback noise, which is more realistic than assuming a uniform noise rate across preference samples.
  2. It combines optimal transport-based noise estimation with a theoretically unbiased surrogate loss, which provides a concrete denoising mechanism and theoretical guarantees.
  3. The improvements extend beyond reward prediction accuracy to downstream RLHF policy safety, supporting the practical value of the proposed approach.
Critical weaknesses
  1. The experiments are conducted on semi-synthetic noise generated using a reward model's predictions. And the noise was defined to be sample-wise. It does not capture the complexity of real human disagreement or LLM-judge errors. It remains unclear how well CleanRM performs on naturally noisy preference data.
  2. This method has strong dependence on the quality of clean feedback. The method may learn to match the clean distribution without recovering the true noise mechanism, particularly when the clean subset is small or unrepresentative.
  3. The method is conditioned on binary pointwise rewards. Its applicability to other common settings, like pairwise preference or continuous rewards, remains unestablished.
  4. There's no reward benchmark accuracy reported.
Questions for authors
  1. How sensitive is CleanRM to the selection and representativeness of the clean subset? Would the method remain effective if the clean feedback were concentrated in certain domains or preference classes rather than sampled randomly?
  2. Can CleanRM still win when the noise generator and the estimator don’t share an architecture? Currently the noise synthesizing uses a reward model’s p(x) and the estimator is the same backbone with a different head. What happens if noise synthesizing is structurally different (e.g., a lexical heuristic, length bias, position bias, or a held-out model family), while keep the estimator fixed?
Reviewer_n53ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CleanRM, which uses a few pieces of clean feedback to calibrate sample-wise noise ratios via optimal transport.

Critical strengths
  1. This paper proposes a novel reward modeling framework that leverages a small amount of clean feedback to calibrate reward models trained on noisy data through optimal transport (OT).
  2. The experimental results demonstrate that the proposed method achieves competitive performance, consistently outperforming several strong baselines across multiple benchmarks.
Critical weaknesses

1. Limited Comparisons with Existing Reward Models.

Although the paper includes several denoising baselines, the experimental evaluation lacks comprehensive comparisons with state-of-the-art reward modeling methods, particularly discriminative and generative reward models. Including these baselines would help clarify the advantages of the proposed framework over existing reward modeling approaches.

2. Limited Methodological Novelty.

The methodological novelty of the proposed framework is somewhat limited, as optimal transport (OT) has been extensively explored in learning with noisy labels. The proposed approach appears to build upon established OT-based distribution alignment and noise correction techniques. Therefore, the authors should further clarify the fundamental methodological contributions that distinguish their framework from existing noise-robust learning methods.

3. Insufficient Experimental Evaluation.

The experimental results are not sufficiently convincing due to the limited diversity of evaluation metrics. Although the proposed method achieves promising performance on several benchmarks, the current evaluation does not comprehensively assess the robustness, reliability, and practical effectiveness of the learned reward models. Additional evaluation metrics and more comprehensive experiments would strengthen the empirical claims.

Questions for authors

1. Relationship to Existing Work.

The authors should further clarify the connections and key differences between the proposed method and the existing work, "Optimal Transport for LLM Reward Modeling from Noisy Preference." In particular, what are the fundamental methodological differences, and what additional contributions does the proposed framework offer beyond this prior work?

2. Scalability and Computational Complexity.

It remains unclear whether the proposed optimal transport (OT) framework can scale efficiently to large-scale training. The authors should provide a more detailed analysis of the additional computational and memory overhead introduced by the OT component, including its impact on training efficiency and scalability. Empirical comparisons of training time and computational costs would further strengthen the practical relevance of the proposed approach.

Reviewer_4grrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the problem of feedback noise in preference data, spanning diverse sources like subjective human judgement, annotation mistakes and imperfect LLM judges. The authors consider reward modelling under sample-wise feedback noise instead of imposing a uniform noise ratio assumption. They propose CleanRM, which aligns the joint distribution of preferences reverted from noisy feedback with that of clean feedback—obtained from expert auditing or repeated annotation—via Wasserstein OT, then uses the estimated noise ratios to construct a noise-corrected reward-modelling loss. They prove that this new objective is unbiased wrt the clean risk; empirically, they demonstrate that CleanRM produces accurate reward signals under noisy feedback and improves downstream RLHF alignment.

Critical strengths
  • S1. Problem identification. This paper addresses an important underexplored problem in reward modelling on noisy preference data. The reviewer agrees that feedback reliability does vary across prompt-reponses pairs, which motivates using more expressive (than uniform) ways of modelling noise rates and learning from differently lossy data.
  • S2. Connecting reward model improvements to downstream model behaviour. The evaluation is meticulous and includes multiple preference datasets, architecture families, model sizes, ablations and downstream GRPO results. The policy safety results makes a case for the practical advantages of CleanRM beyond gains in reward-prediction metrics alone.
Critical weaknesses
  • W1. Distinction between signal and noise is under justified. The method assumes a clean binary preference and regards any deviations as corruption. However, real preference data is pluralistic and disagreements may signal underlying differences in values or context which are important to learn. The authors propose that “clear instructions yield consistent feedback” but have not substantiated why sample-wise variability equals noise: the underlying signal could be (and realistically is) complex and sample-dependent.

  • W2. Potential overfitting to small clean set insufficiently addressed. The noise estimator may likely learn spurious correlations or sampling biases in the calibration set and propagate them through loss correction. The authors consider using 5% of training data as clean feedback---which firstly, may not be practical given the scale of LLM training sets at any given stage—but do not clearly report absolute calibration-set sizes or establish sample-complexity requirements. The clean-set-size sweep and standard regularization do no establish robustness to different realisations of clean sets, e.g. unrepresentative or uninformative samples, missing subgroups, or variation across independently sampled calibration sets.

  • W3. Unclear how sample-wise noise is identified / requires strong assumptions. The authors aggregate highly disparate “noise” sources like chance annotation mistakes, systematic judge bias, genuine preference disagreements into two input-dependent flip probabilities, and do not attempt to distinguish their sources. The reviewer is not convinced that noise estimation remains reliable under correlated feedback or shared source biases. The identification of “noise” here relies on strong local matching and clean-data coverage assumptions; applicability to real preference data remains unclear.

Questions for authors
  • Q1. How does CleanRM interact with GRPO’s reward normalization? Methods like GRPO remove common reward shift and positive rescaling within each prompt group; they have also been shown to significantly benefit from sample-wise variation in response quality. Does using CleanRM with GRPO have neutral, positive or negative effects on GRPO, in terms of e.g. within-group rankings, preserving useful variance? Could inverse noise correction in fact amplify reward-estimation variance when (1−μ(x)−ν(x))(1-\mu(x)-\nu(x)) is small?
  • Q2. How robust is CleanRM to intentional malicious mislabeling? Have you evaluated MNAR noise sources, e.g. targeted label flips, coordinated or correlated annotator biases, or contaminated clean calibration sets? Can CleanRM handle attacks in regions underrepresented by clean feedback, where local corruption may violate the nondegeneracy assumption despite a low overall corruption rate? The jailbreak evaluations are helpful for assessing inference-time robustness but do not provide evidence for training-time poisoning robustness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.