acceptodds
Under review as a conference paper at ICLR 2027

BiasReducer: Adaptive Bias Mitigation for Reward Models

Abstract

Reward models score responses from large language models (LLMs) and guide LLM training toward human preferences. However, reward models can favor superficial attributes such as length or confidence, leading LLMs to produce higher-scoring but not more correct responses. Existing mitigation methods either retrain the reward model or apply a fixed correction to one known bias, such as a preference for longer responses. Retraining requires additional data and computational resources, while existing editing methods require the target bias to be specified in advance and use a fixed edit for that bias. To this end, we propose BiasReducer, a lightweight framework that edits only the linear reward head and selects the relevant edits for each new dataset. First, BiasReducer uses a sparse autoencoder (SAE)-style encoder to learn which attributes (e.g., length and confidence) the reward model is sensitive to. Second, it learns how to reduce the reward model's dependence on each attribute by determining which direction to adjust the reward head and how much to adjust it. Third, for a new dataset, it ranks the attributes by their influence on reward scores, selects the relevant ones, and edits the reward model accordingly. BiasReducer consistently improves reward-model robustness to biases toward superficial response attributes. Across five reward models, BiasReducer-M improves the three benchmarks by 8.3, 18.0, and 6.9 percentage points on average, outperforming the two training-based baselines. The gains transfer downstream, reducing unnecessary verbosity and sycophancy while maintaining comparable judged quality.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward models, reward hacking, model alignment, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gMz2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BIASREDUCER, a lightweight method that reduces reward models’ reliance on superficial attributes such as length, formatting and sycophancy by editing only the linear reward head. It prepares reusable corrections for a set of predefined attributes and, for each new dataset, automatically decides which ones to apply. Across five reward models and three bias benchmarks, the authors claim that it outperforms two finetuning baselines and reduces verbosity and sycophancy in downstream best-of-N selection and RL training.

Critical strengths
  1. The paper proposes a lightweight and effective design, which helps to give more control over reward model ability.
  2. The authors conducted comprehensive experiments beyond evaluation on reward model benchmarks, which further verifies the effectiveness of the design.
  3. The paper is logically clear and easy to follow, with clear motivation.
Critical weaknesses
  1. The core methodology design relies on seven hand-picked attributes, which is heuristic and defined by handcrafted measurements with no justification of coverage. Biases outside the set, e.g. answer logic, specific writing styles etc, are not discussed and handled, which may undermine the claim that the target bias need not be specified in advance. An explanation on how the design covers all possible reward hackings would help (if not, should propose a possible fix or admit as limitation).
  2. The current debiasing step cannot be distinguished from simply flipping the RM toward anti-style preferences. All three benchmarks only contain cases where the less-styled response is preferred, so “always pick the shorter one” scores 100% on Arena-StyleConflict. Experiments on benchmark where style and quality agree (e.g., RM-Bench Easy/Normal) would help. Also, whether the anti-style edits would hurt performance when style is genuinely desired remains unknown (this is also an example of weakness 1, since your designed attributes are not guaranteed to cover all aspects), test on related benchmarks (e.g. Personalized RewardBench) would also help.
  3. The paper misses several key baselines and exists unfair comparison. Simple length penalties, the closest editing methods (e.g., HARVE) and a no-SAE variant are not compared. BiasReducer also selects edits on unlabeled test data, while the fine-tuning baselines see no target data.
Questions for authors

please refer to the weaknesses, especially weakness 2, as it causes directly doubt on your major argument. I will increase my rating if all concerns are addressed.

2 comments

Thank you for your response and additional experiments. My concern regarding W3 has been resolved, but W1 and W2 remain.

Regarding W1: I am still unclear about how the seven hand-picked attributes would help generalize to new tasks where the underlying biases are orthogonal to these attributes. While you report improvements on beauty, sentiment, and concreteness, the underlying mechanism that enables such generalization remains unclear. I would appreciate further discussion of why the proposed approach is expected to generalize beyond the predefined attributes.

Regarding W2: My concern remains that the improvements on the Easy and Medium subsets of RM-Bench are marginal. The model may simply learn to counteract specific biases rather than improve its underlying reward modeling capability. For instance, on Arena-StyleConflict, a heuristic that always favors the shorter response could achieve 100% accuracy. Therefore, improvements on this subset alone do not necessarily demonstrate better reward modeling capabilities. Anyway, I do appreciate your transparency in reporting these results.

AuthorsOpenReview ↗

Further Response to Reviewer gMz2

We thank the reviewer for the thoughtful comment.

1. Regarding W1: I am still unclear about how the seven hand-picked attributes would help generalize to new tasks where the underlying biases are orthogonal to these attributes. While you report improvements on beauty, sentiment, and concreteness, the underlying mechanism that enables such generalization remains unclear. I would appreciate further discussion of why the proposed approach is expected to generalize beyond the predefined attributes.

Thank you for this follow-up. We would like to clarify the distinction between biases not explicitly included in our attribute bank and biases that are truly orthogonal to the learned edit directions.

First, biases outside the predefined categories are not necessarily independent of the seven attributes. BiasReducer learns editing directions in the RM's representation space, rather than directly applying penalties to surface-level text features. Different biases may share underlying representations. For example, beauty may be related to formatting, while sentiment may overlap with confidence. In such cases, editing the reward head along one attribute direction may also reduce reliance on related biases. This offers a possible explanation for the improvements on JBB categories without explicitly corresponding attributes.

Second, truly orthogonal biases are outside the current method's guaranteed coverage. If a bias has no overlap with the learned edit directions, BiasReducer cannot be expected to mitigate it. Our empirical results demonstrate improvements beyond explicitly modeled bias categories, but do not establish universal coverage. The framework supports adding new attributes to address additional biases.

2. Regarding W2: My concern remains that the improvements on the Easy and Medium subsets of RM-Bench are marginal. The model may simply learn to counteract specific biases rather than improve its underlying reward modeling capability. For instance, on Arena-StyleConflict, a heuristic that always favors the shorter response could achieve 100% accuracy. Therefore, improvements on this subset alone do not necessarily demonstrate better reward modeling capabilities. Anyway, I do appreciate your transparency in reporting these results.

We would like to clarify that the primary objective of BiasReducer is to mitigate spurious biases in reward models through lightweight reward-head editing, rather than to improve their general preference modeling capability. Importantly, our method aims to reduce reliance on superficial attributes while largely preserving the reward model's original preference modeling capability. As demonstrated in our additional experiments, BiasReducer maintains nearly unchanged performance on RewardBench 2, with a modest decrease on RM-Bench Easy/Normal, while achieving substantially larger improvements on RM-Bench Hard (+5.9 pp). These results are consistent with our intended objective: mitigating spurious biases without substantially compromising general preference modeling performance. We agree that improvements on bias-conflict benchmarks should not be interpreted as evidence of a general enhancement in reward modeling.

Reviewer_gCX4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work analyzes biases in reward models without retraining the model itself. The proposed method BiasReducer first trains an sparse autoencoder (SAE) with additional loss constraints (behavioral, interventional, and artifact supervision), to encourage designated latent coordinates toward the predefined bias attributes such as length, confidence, and formatting. These learned representations are then used for attribute-specific edits on the reward head. This gives a edit bank that can also be applied onto a new dataset, where BiasReducer selects and combines edits based on unlabeled responses.

Critical strengths
  1. Addresses an important question for reward modeling, which could influence downstream LLM performance
  2. Experiments show a strong empirical improves from the vanilla baseline
  3. Figures and experimental results are clearly presented
Critical weaknesses
  1. Method complexity: BiasReducer introduces multiple hyperparameters and attribute-specific interventions. It is unclear how much tuning / computational overhead may be required in practice, which limits its practicality compared with simpler editing methods
  2. Baselines & related works: A few prior works are cited, the baselines remain limited and missing a few methods with similar approach that finds interpretable features. For example, [1, 2] also use SAE or PCA on embedding differences, [3] is another reward model bias discovery method. A direct comparison and clearer discussion of these works are appropriate to distinguish BiasReducer’s novelty

References

[1] What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data
[2] Rethinking Diverse Human Preference Learning through Principal Component Analysis
[3] Automatically Finding Reward Model Biases

Questions for authors
  1. While Table 2 show partial results for quality, verbosity, and sycophancy, how does editing affects other designated target attributes mentioned?
  2. For Sec 3.2 / Appendix B2, deterministic and rule-based interventions were mentioned. Could the authors clarify the concrete definitions and rules for each attribute? Their construction seems central since these signals directly determine what the learned SAE coordinates represent.
Reviewer_N9DMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BiasReducer, a post-hoc method for mitigating response-attribute biases in reward models without retraining the full model. It learns SAE-style representations aligned with predefined attributes using preference contrasts and attribute-specific supervision. It then constructs attribute-specific reward-head edits, estimating the edit directions from source preference data and selecting edit strengths under a source-validation accuracy constraint. For a new dataset, BiasReducer ranks candidate edits using unlabeled responses, their hidden representations, and the original reward scores, without access to target preference labels. BiasReducer-S applies a single edit, while BiasReducer-M combines multiple edits. The experiments cover five reward models and three benchmarks. Relative to the original reward models, BiasReducer-M improves average pairwise preference accuracy by 8.3 percentage points on RM-Bench-Hard, 18.0 points on Arena-StyleConflict, and 6.9 points on JudgeBiasBench. The paper also evaluates the edited reward model in Best-of-N selection and GRPO training, reporting reduced unnecessary verbosity and sycophancy, with judged response quality remaining broadly comparable in the tested settings. The main contribution is a framework that separates reusable, source-side attribute editing from target-specific edit selection without requiring target preference labels.

Critical strengths
  1. The separation between edit construction and target-specific selection is practically useful. BiasReducer constructs attribute-specific edits from source preference data and selects among them using unlabeled target responses. Since only the linear reward head is modified, the method offers a relatively lightweight alternative to retraining the full reward model. This makes the approach potentially useful when target preference annotations or additional training resources are unavailable.
  2. The evaluation provides evidence across multiple models and settings. The experiments cover five reward models and three bias-focused benchmarks, and both proposed variants outperform the original models across the reported model–benchmark combinations. The downstream Best-of-N and GRPO experiments also examine whether the changes in reward-model behavior affect response generation, rather than relying exclusively on offline preference accuracy.
Critical weaknesses
  1. The evaluation does not establish how well target-specific edits generalize beyond the response pool used to select them. Appendix A.4 explicitly describes a transductive setting: the method selects edits using the available unlabeled responses from each benchmark and evaluates the edited reward head on that same benchmark. This is not target-label leakage, and the protocol is reasonable when the full target response pool is available before deployment. However, it does not establish that the selected edits will remain effective on unseen or subsequently arriving responses. The response-pool-size analysis examines the stability of edit selection, but agreement with the full-pool decision is not equivalent to evaluation on an independent test set. This limits the evidence for applying the method beyond the specific target pools studied.
  2. The experiments do not fully isolate the contribution of the proposed selection mechanism from that of reward-head editing itself. The paper already includes useful ablations on semantic supervision, edit direction, edit composition, and fixed versus dataset-specific selection. The remaining gap is more specific: there is no direct comparison with simpler post-hoc reward-head editing or concept-erasure approaches under the same source-validation accuracy constraint. In addition, the target-selection signal based on the magnitude of reward-score changes measures how strongly an edit perturbs scores, but does not by itself show that dependence on the intended attribute has decreased while useful preference information is preserved. A more direct analysis of residual attribute dependence would help validate this selection rule. The absence of uncertainty estimates also makes it difficult to assess the stability of smaller gains for individual model–benchmark combinations.
Questions for authors
  1. The target-selection rule combines the association between the original reward and an attribute activation with the magnitude of the reward-score changes induced by an edit. What evidence shows that the score-change signal tracks a reduction in dependence on the intended attribute, rather than merely the magnitude of a perturbation? Can the authors report pre-edit and post-edit attribute–reward relationships and examine how well the ranking signals predict actual edit effectiveness? The answer would materially affect my confidence in the proposed selection mechanism.
  2. Can the authors separate the target response pool used for edit selection from the responses used for final evaluation? Results on an independent set of responses would clarify whether the improvements persist beyond the pool observed during selection.
  3. Can the authors compare BiasReducer against simpler post-hoc reward-head editing or concept-erasure baselines under the same source-validation accuracy constraint, and report variability or uncertainty estimates for the main results? This would help determine whether the gains are attributable to the proposed framework and how robust they are across runs and benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.