- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BIASREDUCER, a lightweight method that reduces reward models’ reliance on superficial attributes such as length, formatting and sycophancy by editing only the linear reward head. It prepares reusable corrections for a set of predefined attributes and, for each new dataset, automatically decides which ones to apply. Across five reward models and three bias benchmarks, the authors claim that it outperforms two finetuning baselines and reduces verbosity and sycophancy in downstream best-of-N selection and RL training.
Critical strengths
- The paper proposes a lightweight and effective design, which helps to give more control over reward model ability.
- The authors conducted comprehensive experiments beyond evaluation on reward model benchmarks, which further verifies the effectiveness of the design.
- The paper is logically clear and easy to follow, with clear motivation.
Critical weaknesses
- The core methodology design relies on seven hand-picked attributes, which is heuristic and defined by handcrafted measurements with no justification of coverage. Biases outside the set, e.g. answer logic, specific writing styles etc, are not discussed and handled, which may undermine the claim that the target bias need not be specified in advance. An explanation on how the design covers all possible reward hackings would help (if not, should propose a possible fix or admit as limitation).
- The current debiasing step cannot be distinguished from simply flipping the RM toward anti-style preferences. All three benchmarks only contain cases where the less-styled response is preferred, so “always pick the shorter one” scores 100% on Arena-StyleConflict. Experiments on benchmark where style and quality agree (e.g., RM-Bench Easy/Normal) would help. Also, whether the anti-style edits would hurt performance when style is genuinely desired remains unknown (this is also an example of weakness 1, since your designed attributes are not guaranteed to cover all aspects), test on related benchmarks (e.g. Personalized RewardBench) would also help.
- The paper misses several key baselines and exists unfair comparison. Simple length penalties, the closest editing methods (e.g., HARVE) and a no-SAE variant are not compared. BiasReducer also selects edits on unlabeled test data, while the fine-tuning baselines see no target data.
Questions for authors
please refer to the weaknesses, especially weakness 2, as it causes directly doubt on your major argument. I will increase my rating if all concerns are addressed.
2 comments
Further Response to Reviewer gMz2
We thank the reviewer for the thoughtful comment.
1. Regarding W1: I am still unclear about how the seven hand-picked attributes would help generalize to new tasks where the underlying biases are orthogonal to these attributes. While you report improvements on beauty, sentiment, and concreteness, the underlying mechanism that enables such generalization remains unclear. I would appreciate further discussion of why the proposed approach is expected to generalize beyond the predefined attributes.
Thank you for this follow-up. We would like to clarify the distinction between biases not explicitly included in our attribute bank and biases that are truly orthogonal to the learned edit directions.
First, biases outside the predefined categories are not necessarily independent of the seven attributes. BiasReducer learns editing directions in the RM's representation space, rather than directly applying penalties to surface-level text features. Different biases may share underlying representations. For example, beauty may be related to formatting, while sentiment may overlap with confidence. In such cases, editing the reward head along one attribute direction may also reduce reliance on related biases. This offers a possible explanation for the improvements on JBB categories without explicitly corresponding attributes.
Second, truly orthogonal biases are outside the current method's guaranteed coverage. If a bias has no overlap with the learned edit directions, BiasReducer cannot be expected to mitigate it. Our empirical results demonstrate improvements beyond explicitly modeled bias categories, but do not establish universal coverage. The framework supports adding new attributes to address additional biases.
2. Regarding W2: My concern remains that the improvements on the Easy and Medium subsets of RM-Bench are marginal. The model may simply learn to counteract specific biases rather than improve its underlying reward modeling capability. For instance, on Arena-StyleConflict, a heuristic that always favors the shorter response could achieve 100% accuracy. Therefore, improvements on this subset alone do not necessarily demonstrate better reward modeling capabilities. Anyway, I do appreciate your transparency in reporting these results.
We would like to clarify that the primary objective of BiasReducer is to mitigate spurious biases in reward models through lightweight reward-head editing, rather than to improve their general preference modeling capability. Importantly, our method aims to reduce reliance on superficial attributes while largely preserving the reward model's original preference modeling capability. As demonstrated in our additional experiments, BiasReducer maintains nearly unchanged performance on RewardBench 2, with a modest decrease on RM-Bench Easy/Normal, while achieving substantially larger improvements on RM-Bench Hard (+5.9 pp). These results are consistent with our intended objective: mitigating spurious biases without substantially compromising general preference modeling performance. We agree that improvements on bias-conflict benchmarks should not be interpreted as evidence of a general enhancement in reward modeling.
Thank you for your response and additional experiments. My concern regarding W3 has been resolved, but W1 and W2 remain.
Regarding W1: I am still unclear about how the seven hand-picked attributes would help generalize to new tasks where the underlying biases are orthogonal to these attributes. While you report improvements on beauty, sentiment, and concreteness, the underlying mechanism that enables such generalization remains unclear. I would appreciate further discussion of why the proposed approach is expected to generalize beyond the predefined attributes.
Regarding W2: My concern remains that the improvements on the Easy and Medium subsets of RM-Bench are marginal. The model may simply learn to counteract specific biases rather than improve its underlying reward modeling capability. For instance, on Arena-StyleConflict, a heuristic that always favors the shorter response could achieve 100% accuracy. Therefore, improvements on this subset alone do not necessarily demonstrate better reward modeling capabilities. Anyway, I do appreciate your transparency in reporting these results.