Mitigating Cognitive Bias in RLHF by Altering Rationality

How can we make models robust to even imperfect human feedback? In reinforcement learning from human feedback (RLHF), human preferences over model outputs are used to train a reward model that assigns scalar values to responses. Because these rewards are inferred from pairwise comparisons, this learning depends on an assumed relationship between latent reward differences and observed preferences, typically modeled using a Boltzmann formulation in which a rationality parameter $\beta$ informs how consistently preferences reflect these reward differences. In practice, $\beta$ is typically treated as a fixed constant that assumes uniform annotator reliability. However, human feedback is not actually this simplistic: real human judgments are shaped by cognitive biases, leading to systematic deviations from reward-consistent behavior that arise in context. To address this, we treat rationality as context- and annotation-dependent. We design an approach to dynamically adjust the rationality parameter $\beta$ during reward learning by effectively downweighting comparisons that likely reflect biased or unreliable judgments. Empirically, we show this approach learns a useful and more rational downstream model, even when finetuning on datasets with strongly biased preferences.