- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes making reward learning more robust to cognitive bias by using an LLM judge to score how biased each preferred answer is, then modifying the reward-model objective so potentially biased preferences have less influence. The authors construct preference data from BRU, with 205 questions covering eight cognitive biases, and CogBias, with approximately 29,000 cases covering 29 biases, deliberately labeling the biased answer as preferred 75% of the time by default. They train reward models on these preferences and use them for GRPO policy finetuning. On held-out CogBias questions, the method achieves 67.6% unbiased-answer accuracy, compared with 1.8% for standard training on the same corrupted data and 52.4% for filtering out highly biased comparisons. The main contribution is a simple way to incorporate judge-estimated feedback reliability into reward learning, demonstrated in controlled synthetic-bias settings.
Critical strengths
- The paper's intervention method is clear and simple. The bias judged by an LLM judge simply supplies as a scaling factor on the reward score difference between chosen and rejected samples. This design makes usage of existing preference datasets quite convenient.
- The paper did quite comprehensive ablation studies. The paper tested the main method against other ways of selecting beta (fixed and random) and against directly dropping the biased samples. They also tested on different models and different bias judges.
Critical weaknesses
- I find the bias/task targeted by this paper limited in scope. This paper's training and primary evaluation use binary-choice cognitive-bias benchmarks with artificial preference corruption. I acknowledge that this is a type of bias made by human annotator, but this contributes a narrow domain. Many papers have report in the field on annotator biases regarding gender, races, or political opinion. It would be more empirically strengthened if the method could be shown effective on other forms of such biases that are commonly seen in preference data.
- I appreciate the comprehensive baselines the authors already provided, but I find 2 types of baselines critical for establishing paper's method: 1) weighting the entire loss by beta (outside of the log sigmoid) rather than just the reward difference, how does this compare? 2) training with the normal BT loss but ask the bias judge to swap the chosen and rejected labels if the judge is confident that this is biased. I put weak rejection for overall at this stage but I will consider moving to accept if these 2 points are addressed adequately.
Questions for authors
Q1: See weakness 1. Could the authors perform additional experiments to show the method's effectiveness on other forms of biases? Q2: See weakness 2. Could the authors add the baselines or explain why they think is not necessary?
3 comments
As we near the end of the discussion period, would the reviewers please take a look at our responses and consider updating their scores if their feedback has been adequately addressed?
Thank you for clarifying and thanks for the additional experiments. These experiments partially addressed my concerns. I still have a remaining concern of limited task scope. You explained that the stereotype data also comes from CogBias, the dataset used in main study but whether these generalize to biases in other widely used preference datasets like HHRLHF etc. remains unclear. Therefore, I will keep my rating.
Additional Comparisons
Training with BT loss but swapping chosen and rejected labels if judge is confident it is biased >0.5 Label-swapping suffers from a key challenge - what if you have a preference where both options are biased? To show the limitation of the label-swapping method, we added a new subset dataset from CogBias (30 pairs per type of bias) which replaces the pairs where cognitively biased options were chosen over ground truth ones with ones where there is a more and a less cognitively biased option. This represents the situation where there are multiple cognitively biased options. While the debiased method is able to successfully down-weight them, the label swapping method is unable to recover from swapping to alternate still biased preferences. As a result, label-swapping is flawed as an alternative to the debiased method.
Weighting the entire loss by beta (outside of the log sigmoid): There is a minor improvement in using our debiased method over the beta-weighted loss. Though the beta-weighted loss and our debiased method are quite similar, this is expected since it is a minor translation of the beta that is still being dynamically modulated. We choose to use it in the location traditionally attributed to a beta parameter to match with prior art and the theory behind what beta represents as a rationality parameter Jeon et al, 2020.
Given these additional baselines, as well as the demonstration of the method working on ethnic, gender, and religious biases, would you consider raising your score for the paper?