- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CleanRM, a framework for reward modeling under sample-wise feedback noise. It uses a few clean feedback to calibrate sample-wise noise ratios. It constructs a noise-corrected surrogate loss that is theoretically unbiased with the clean risk. Experiments demonstrate improved reward modeling performance over existing baselines, and downstream GRPO experiments show consistent improvements in policy safety.
Critical strengths
- The paper explicitly deals with sample-wise heterogeneous feedback noise, which is more realistic than assuming a uniform noise rate across preference samples.
- It combines optimal transport-based noise estimation with a theoretically unbiased surrogate loss, which provides a concrete denoising mechanism and theoretical guarantees.
- The improvements extend beyond reward prediction accuracy to downstream RLHF policy safety, supporting the practical value of the proposed approach.
Critical weaknesses
- The experiments are conducted on semi-synthetic noise generated using a reward model's predictions. And the noise was defined to be sample-wise. It does not capture the complexity of real human disagreement or LLM-judge errors. It remains unclear how well CleanRM performs on naturally noisy preference data.
- This method has strong dependence on the quality of clean feedback. The method may learn to match the clean distribution without recovering the true noise mechanism, particularly when the clean subset is small or unrepresentative.
- The method is conditioned on binary pointwise rewards. Its applicability to other common settings, like pairwise preference or continuous rewards, remains unestablished.
- There's no reward benchmark accuracy reported.
Questions for authors
- How sensitive is CleanRM to the selection and representativeness of the clean subset? Would the method remain effective if the clean feedback were concentrated in certain domains or preference classes rather than sampled randomly?
- Can CleanRM still win when the noise generator and the estimator don’t share an architecture? Currently the noise synthesizing uses a reward model’s p(x) and the estimator is the same backbone with a different head. What happens if noise synthesizing is structurally different (e.g., a lexical heuristic, length bias, position bias, or a held-out model family), while keep the estimator fixed?