acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Cognitive Bias in RLHF by Altering Rationality

Abstract

How can we make models robust to even imperfect human feedback? In reinforcement learning from human feedback (RLHF), human preferences over model outputs are used to train a reward model that assigns scalar values to responses. Because these rewards are inferred from pairwise comparisons, this learning depends on an assumed relationship between latent reward differences and observed preferences, typically modeled using a Boltzmann formulation in which a rationality parameter informs how consistently preferences reflect these reward differences. In practice, is typically treated as a fixed constant that assumes uniform annotator reliability. However, human feedback is not actually this simplistic: real human judgments are shaped by cognitive biases, leading to systematic deviations from reward-consistent behavior that arise in context. To address this, we treat rationality as context- and annotation-dependent. We design an approach to dynamically adjust the rationality parameter during reward learning by effectively downweighting comparisons that likely reflect biased or unreliable judgments. Empirically, we show this approach learns a useful and more rational downstream model, even when finetuning on datasets with strongly biased preferences.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
RLHF, cognitive bias, human preferences, rationality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SRkbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes making reward learning more robust to cognitive bias by using an LLM judge to score how biased each preferred answer is, then modifying the reward-model objective so potentially biased preferences have less influence. The authors construct preference data from BRU, with 205 questions covering eight cognitive biases, and CogBias, with approximately 29,000 cases covering 29 biases, deliberately labeling the biased answer as preferred 75% of the time by default. They train reward models on these preferences and use them for GRPO policy finetuning. On held-out CogBias questions, the method achieves 67.6% unbiased-answer accuracy, compared with 1.8% for standard training on the same corrupted data and 52.4% for filtering out highly biased comparisons. The main contribution is a simple way to incorporate judge-estimated feedback reliability into reward learning, demonstrated in controlled synthetic-bias settings.

Critical strengths
  1. The paper's intervention method is clear and simple. The bias judged by an LLM judge simply supplies as a scaling factor on the reward score difference between chosen and rejected samples. This design makes usage of existing preference datasets quite convenient.
  2. The paper did quite comprehensive ablation studies. The paper tested the main method against other ways of selecting beta (fixed and random) and against directly dropping the biased samples. They also tested on different models and different bias judges.
Critical weaknesses
  1. I find the bias/task targeted by this paper limited in scope. This paper's training and primary evaluation use binary-choice cognitive-bias benchmarks with artificial preference corruption. I acknowledge that this is a type of bias made by human annotator, but this contributes a narrow domain. Many papers have report in the field on annotator biases regarding gender, races, or political opinion. It would be more empirically strengthened if the method could be shown effective on other forms of such biases that are commonly seen in preference data.
  2. I appreciate the comprehensive baselines the authors already provided, but I find 2 types of baselines critical for establishing paper's method: 1) weighting the entire loss by beta (outside of the log sigmoid) rather than just the reward difference, how does this compare? 2) training with the normal BT loss but ask the bias judge to swap the chosen and rejected labels if the judge is confident that this is biased. I put weak rejection for overall at this stage but I will consider moving to accept if these 2 points are addressed adequately.
Questions for authors

Q1: See weakness 1. Could the authors perform additional experiments to show the method's effectiveness on other forms of biases? Q2: See weakness 2. Could the authors add the baselines or explain why they think is not necessary?

3 comments
AuthorsOpenReview ↗

Additional Comparisons

Training with BT loss but swapping chosen and rejected labels if judge is confident it is biased >0.5>0.5 Label-swapping suffers from a key challenge - what if you have a preference where both options are biased? To show the limitation of the label-swapping method, we added a new subset dataset from CogBias (30 pairs per type of bias) which replaces the pairs where cognitively biased options were chosen over ground truth ones with ones where there is a more and a less cognitively biased option. This represents the situation where there are multiple cognitively biased options. While the debiased method is able to successfully down-weight them, the label swapping method is unable to recover from swapping to alternate still biased preferences. As a result, label-swapping is flawed as an alternative to the debiased method.

debiasedlabel-swapped
BRU dataset49.2%47%
CogBias dataset67.6%94.9%
biasedPairs dataset67.8%37.7%

Weighting the entire loss by beta (outside of the log sigmoid): There is a minor improvement in using our debiased method over the beta-weighted loss. Though the beta-weighted loss and our debiased method are quite similar, this is expected since it is a minor translation of the beta that is still being dynamically modulated. We choose to use it in the location traditionally attributed to a beta parameter to match with prior art and the theory behind what beta represents as a rationality parameter Jeon et al, 2020.

debiasedbeta-weighted
BRU dataset49.2%42.8%
CogBias dataset67.6%66.5%

Given these additional baselines, as well as the demonstration of the method working on ethnic, gender, and religious biases, would you consider raising your score for the paper?

As we near the end of the discussion period, would the reviewers please take a look at our responses and consider updating their scores if their feedback has been adequately addressed?

Thank you for clarifying and thanks for the additional experiments. These experiments partially addressed my concerns. I still have a remaining concern of limited task scope. You explained that the stereotype data also comes from CogBias, the dataset used in main study but whether these generalize to biases in other widely used preference datasets like HHRLHF etc. remains unclear. Therefore, I will keep my rating.

Reviewer_qieNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses cognitive biases in RLHF. The authors argue that the conventional BT reward model assumes uniform annotator rationality through a fixed parameter β\beta, which may fail when human preferences are systematically affected by cognitive biases. They propose an instance-dependent rationality parameter computed from an LLM-based bias detector. Comparisons considered more susceptible to cognitive bias receive lower β\beta values during reward-model training, followed by GRPO. The method is evaluated on the BRU and CogBias datasets, where cognitively biased preference labels are simulated at different frequencies. The experiments report substantial improvements in ground-truth accuracy over fixed-β\beta and random-β\beta baselines, including results across Mistral-7B, Qwen3-8B, and Phi-4.

Critical strengths

The paper addresses the important problem of learning from cognitively biased human feedback and demonstrates substantial improvements across different bias conditions and model architectures.

Critical weaknesses

The bias detector provides strong correctness-related supervision, making it difficult to attribute the improvements specifically to dynamic rationality modeling. Moreover, reducing β\beta inside the BT sigmoid does not necessarily downweight gradient contributions, leaving the proposed mechanism insufficiently justified.

Questions for authors
  1. How does dynamic β\beta compare with detector-based relabeling or explicit sample weighting?

  2. Does reducing β\beta actually downweight biased comparisons during training, and does the method generalize to real human feedback?

Reviewer_Yrt2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how cognitive biases can affect reward modeling and downstream RLHF. The authors propose a dynamic rationality modeling approach that uses an LLM-based bias detector to estimate the reliability of each preference annotation and adjusts the Bradley–Terry rationality parameter accordingly, reducing the influence of potentially biased feedback during reward model training. Experiments on two cognitive-bias datasets show that the proposed method substantially improves robustness to biased preference annotations and downstream policy accuracy.

Critical strengths
  1. Annotation bias and noise exist in preference datasets and influence the final RLHF. This paper investigates a meaningful question.

  2. The writing is clear and easy to follow. The proposed method is also easy to implement and incorporate into the RLHF pipeline.

Critical weaknesses

The method uses an additional LLM judge to estimate whether a preference label is biased and then reduces its influence during reward model training, which raises a critical question that, if we can classify the preference label, why not clean the dataset and then train the reward model? Therefore, I think more direct baselines such as confidence-weighted BT loss, hard filtering or label correction using the same bias detector are necessary. Without these comparisons, it is difficult to determine how much of the improvement comes specifically from the proposed rationality parameterization.

Besides, since the estimated bias score directly determines how strongly each preference contributes to reward model training, inaccurate or poorly calibrated judgments could potentially suppress useful preference signals. Although the paper investigates robustness to noisy judging, the detector also benefits from bias-type-specific calibration, and the mapping parameters are selected using validation data. It would be helpful to better understand how well the method generalizes to unseen bias types or preference datasets where such calibration information is unavailable.

By the way, there are lots of debiased reward model studies. I think it would be better to include these works in the related work and have an experimental comparison.

Questions for authors

Please refer to the Weaknesses.

Comments on the paper

3 comments

Thanks for your response.

For the data filtering baseline, it would be helpful to provide more implementation details in Section 4.5, such as the proportion of filtered samples, the filtering threshold, and the resulting training data size. Currently, the paper only reports the final accuracy values, which makes it difficult to understand the comparison.

Since your method already obtains cognitive-bias scores from an additional LLM judge, I do not see why these scores cannot also be used by existing reward-model debiasing methods. For example, DIR [1] explicitly incorporates known bias attributes into reward model training through an information-theoretic regularization term. In your setting, the bias detector already provides the information needed to construct such attributes, so this seems to be a natural baseline. Even simpler alternatives, such as a correlation-based penalty (like Pearson correlation) or confidence-weighted BT loss, could be implemented using the same detector outputs. Regarding unsupervised debiasing, InfoRM [2] achieves effectively debiased results without explicitly using bias labels. My concern is not whether the proposed dynamic rationality parameterization can outperform fixed-beta training, but whether it provides advantages over existing debiasing methods when all methods have access to the same bias information.

[1] Li et al., Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance, ICLR 2026. https://arxiv.org/abs/2512.23461

[2] Miao et al., InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling, NeurIPS 2024. https://arxiv.org/abs/2402.09345

Thanks for the authors' response. I would encourage the authors to deepen the technical analysis, strengthen the experimental evaluation, and clarify the presentation. At this stage, my overall score remains unchanged.

AuthorsOpenReview ↗

Summary of Discussion Period

Summary Reviewers found the question being addressed important (R1, R2), the method demonstrated improvements across conditions and biases (R2, R3), and noted the clarity of the writing and method (R1, R3). We have responded to every concern raised by the reviewers and run additional baselines to show the impact of our method.

Concerns

Additional hard filtering and label correcting baselines using the bias detector (R1, R2, R3): We have provided additional information about hard filtering (see response to R1) and added a label-correcting baseline (see response to R2). We also compared with the beta-weighted loss (see response to R3).

Bias type calibration (R1): The BRU dataset already does not have bias type calibration.

Debiased reward model (R1): We implemented the DIR baseline, which our method improves over by 38.9% on the BRU dataset (see response to R1).

Is it realistic to human feedback? (R2) By showing the method is robust to varying levels of error in the human feedback (as shown in section 4.3), we show the method could generalize to real human feedback, regardless of how much error is present.

Does reducing beta actually down-weight biased comparisons during training? (R2) We show mathematical analysis showing this is directionally true (see response to R2).

Can it generalize to other kinds of bias? (R3) On a subset that deals with social biases like gender and ethnicity, our debiased method achieved a 99.1% ground-truth rate, compared with 43.1% for the untuned model and 0.9% for the standard BT model (β = 1.0). See response to R3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.