- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies counterfactual invariance in open-ended generation, where a response should not change with demographic information that is causally irrelevant to the task. Each counterfactual prompt pair is labeled invariant, dependent, or agnostic, so invariance is required only where the attribute does not change what a correct answer contains. The paper first measures the reward model directly. Scoring an identical response under swapped prompts, the Skywork reward model favors the white or male prompt in 69.0% and 73.8% of comparisons, and the preference is larger for responses sampled under those prompts (Tab. 1). Within-prompt normalization in GRPO cancels the constant part of this preference but not the own-prompt bonus, so helpfulness-only training rewards the policy for tailoring responses to each arm. The paper proposes four objectives that connect the two arms, and the final one compares both sets of responses under the same prompt so that prompt-specific reward offsets cancel. It builds 2,582 counterfactual pairs and 725 demographic-free prompts from four public corpora. Experiments post-train Qwen3-4B-Base for 1000 steps on four folds and evaluate on DiscrimEval, Difference Awareness, BBQ, and a four-rubric LLM judge with three judge models. Transformed Reward Expectation Matching reports the highest judge scores and reduces the reward gap between the arms from +3.24 under vanilla GRPO to between +0.41 and +0.97.
Critical strengths
-
The paper measures the reward model rather than assuming it is biased. It scores the same response under both prompts, so any difference comes from the prompt alone (App. A). This identifies the mechanism behind the paper's central claim and does not depend on the LLM judge.
-
The final objective rests on an identification argument rather than an empirical choice. Reward models fix scores only up to a prompt-specific constant, so comparing the two arms directly confounds a difference in behavior with a difference in reward scale. Scoring both sets of responses under the same prompt removes that constant.
Critical weaknesses
-
The comparison basis is too narrow to support the claims. All experiments use one policy model and one reward model. The amplification mechanism is measured as a property of Skywork (App. A), so neither the amplification nor the effectiveness of the mitigations is established for any other reward model. The only training baseline is vanilla GRPO. Section 2 cites prior mitigation methods, including BiasDPO (Allam, 2024), BiasGRPO (Reddy et al., 2026), consistency training (Irpan et al., 2025), and causal reward models (Wang et al., 2025b), and none is evaluated. Response Matching resembles the consistency-training family, but it is the paper's own construction rather than a published method run under its reported settings. The results therefore establish improvement over unmitigated training in a single configuration.
-
The headline improvement rests on a measure that cannot be checked. The four-rubric LLM judge carries the main claim, and the rubrics and class labels will not be released (Sec. 5 and the Ethics Statement). Inter-judge agreement is fair, with mean pairwise quadratic-weighted between 0.260 and 0.386 (Tab. 6). No numerical results or measures of variability are reported anywhere, only training curves. The public benchmarks do not corroborate the claim. BBQ does not separate the methods (App. I.1), and the DiscrimEval gap grows under every objective and ends far above the initializer (Sec. 5). Difference Awareness is the only public benchmark that separates the methods.
Questions for authors
Clarification requests:
- Report a table of numerical results for all methods and benchmarks at a stated checkpoint, with variability across folds.
- State whether the rubrics and the class labels will be released. The main claim rests on them and cannot be verified otherwise.
- State what the agreement in Tab. 6 implies for the differences between methods in Fig. 4, given that lies between 0.260 and 0.386.
- Reconcile the claim of improved invariance with BBQ not separating the methods and with the DiscrimEval gap growing under every objective.
- Transformed Reward Expectation Matching answers fewer difference-required Difference Awareness items than vanilla GRPO and Reward Symmetrization under mixed training (App. H.2 and H.5). State the trade-off this implies between invariance and appropriate differentiation.
- All invariance measures here are behavioral, while App. A shows the reward model's preference persists regardless of the policy's output. Prior work reports fair surface behavior alongside retained internal bias [Himelstein et al., Silenced Biases: The Dark Side LLMs Learned to Refuse, AAAI 2026]. Can the proposed objectives distinguish an invariant policy from one that suppresses the surface expression of that preference?
Requested experiments:
- Report the main comparison with at least one additional policy model and one additional reward model.
- Report the measurement of App. A for those additional reward models, since the own-prompt bonus is what drives the amplification.
- Compare against the prior mitigation methods cited in Sec. 2, such as BiasDPO (Allam, 2024), BiasGRPO (Reddy et al., 2026), and consistency training (Irpan et al., 2025), or state why they are not comparab
9 comments
Mixed training trade-off. We agree that this is a trade-off. In mixed training, TREM changes the reward only on invariant pairs, and dependent pairs receive the standard reward. Learning to ignore demographics on invariant prompts likely makes the model somewhat less willing to treat groups differently, which lowers its accuracy on Difference Awareness questions that require different treatment. In open-ended answers to dependent prompts, this effect does not appear. TREM’s dependent judge score stays near the initializer while vanilla GRPO’s drops, and TREM’s utilization score stays similar to the other methods and above the initializer (Appendix I.2).
Invariance versus suppression. Himelstein et al. study safety-aligned models whose refusals hide biases. In our setting, we always start from a base model without safety alignment and train only for helpfulness, so our models do not learn to refuse. Beyond the sampled text, DiscrimEval evaluates the model on its log-probabilities, so it measures the model’s underlying preference and not only its wording. We do not aim to remove demographic information from the model. We want the model to use it only when the task requires it, as in our dependent prompts.
Response to rebuttal
Thank you for the response and for the additional experiments. The second policy and reward model address the generalization concern, and the numerical tables address the missing results. The comparison basis remains thin, as only one prior method was evaluated while three others were declined despite being feasible. A second concern is the decision not to release the labels and rubric. They set both what is trained on and how it is scored, and neither can be examined. The stated reason is that publishing such a judgment would turn it into a de facto standard, yet releasing it is the standard in the field. BBQ, DiscrimEval and Difference Awareness release the labels encoding comparable judgments, and First-Person Fairness publishes the rating prompt of its LLM judge, all four being works this paper cites. My score therefore remains unchanged.
We thank the reviewer for the detailed comments. We tried to address the concern by adding new additional experiments as requested. Here are the answers for the questions:
Second policy and reward model. We retrain vanilla GRPO and TREM with the same setup, once with Phi-4-mini-instruct as the policy and once with AllenAI Llama-3.1-8B-Instruct-RM-RB2 as the reward model. With Phi-4-mini, TREM lowers the DiscrimEval gap and raises Difference Awareness and the judge score at every α (Appendix K.3, Table 10). With RB2, TREM lowers the DiscrimEval gap and raises the judge score at every α (Appendix K.2, Figure 18). In Appendix A, Table 3, we also show that this RB2 reward model exhibits a preference for white/male prompts (where the contents are otherwise identical) to receive higher rewards.
Baselines. We added BiasGRPO, which trains with GRPO on a reward model for unbiased responses (Appendix K.1). Its reward alone collapses helpfulness, and mixed with the Skywork reward it does not lower the DiscrimEval gap. We did not run the other methods. BiasDPO needs a biased and an unbiased completion per prompt, which our open-ended prompts lack. Consistency training is an SFT method, and our SFT initializer already shows low bias, while most of the bias appears during RL post-training (Figure 3). Causal reward models retrain the reward model, which our setting avoids, since we keep the existing reward model fixed.
Numerical results. We used curves because our claim is about how bias grows during training, which a single checkpoint hides. We added results for all methods with mean ± standard deviation across the folds in Tables 1, 2, and 10.
Release of rubrics and labels. We release the counterfactual prompt pairs, but not the class labels, the annotation policy or the rubrics (the rubrics and policies contain labeled few-shot examples which imply the authors’ choices on when demographics are relevant). Deciding when demographics should change an answer is a normative choice, and a released label set would likely become a standard that later work treats as ground truth, which is not the contribution we want this paper to be remembered for. Users can supply their own labels and run the same method. Appendix I.3 supports this. On agnostic pairs, where either label is reasonable, our mitigations score higher than vanilla GRPO under both rubrics.
However, for starting inspiration, the content of our rubrics can be approximated by the names of the judge axes: substance invariance, level capabilities, tonal neutrality, no unsolicited commentary, judiciousness, tonal appropriateness, and utilization.
Agreement Scores of Judges. The κ values lie between 0.260 and 0.386 because the judges often give the top score of 4 out of 4. Since two judges are likely to both give a 4 by chance, κ stays small even when they agree. On the other hand, all three judges give exactly the same score on 65% to 81% of ratings. This is why we report the median score of the three judges in all plots and tables, which reduces the noise from any single judge. Each point in Figure 4 also averages many ratings, so a few disagreements do not change the overall results.
Reconciling the invariance claim. We claim that the objectives reduce the bias that RL adds, not that they remove it, and Section 5 already states that no objective returns the model to the invariance of its initializer. Under invariant-only training, vanilla GRPO ends with the largest DiscrimEval gap and every objective lowers it (Table 1). Under mixed training, all methods end at the same gap, as Appendix I.1 reports. BBQ does not separate the methods because every run, including vanilla GRPO, stays below the initializer’s ambiguous bias, while vanilla GRPO lowers all four open-ended rubric scores. Forced-choice benchmarks miss these failures, which is why we propose an open-ended evaluation method.