acceptodds
Under review as a conference paper at ICLR 2027

Aligning Language Models towards Counterfactual Invariance

12 views

Abstract

Small variations in prompt characteristics can meaningfully alter the semantics, tone, and contextual assumptions of large language model outputs. Such characteristics can involve group variables such as demographics or other sensitive latent attributes. Sensitivity to such input variables is undesirable when demographic information is causally irrelevant to the task, but might still be relevant for generation of semantically coherent responses. This makes the task of enforcing counterfactual invariance while preserving model utility difficult. To study this problem, we first show that standard model alignment recipes (such as reinforcement learning) increase differences in response quality across demographics, while improving overall response quality, ultimately amplifying what is known as counterfactual bias. To address this, we develop a policy gradient technique involving a group level reward function that operates on matched pairs of counterfactual prompts. For completeness, we build a dataset of controlled counterfactual prompt pairs and design an evaluation that separately measures cases where demographic information is causally relevant to the answer. Finally, we validate our findings on several fairness benchmarks. We demonstrate that our methods reduce counterfactual differences while preserving overall response quality and that the models continue to adapt appropriately when such sensitive information is relevant.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment, Counterfactual Invariance, Reinforcement Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BSuKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies counterfactual invariance in open-ended generation, where a response should not change with demographic information that is causally irrelevant to the task. Each counterfactual prompt pair is labeled invariant, dependent, or agnostic, so invariance is required only where the attribute does not change what a correct answer contains. The paper first measures the reward model directly. Scoring an identical response under swapped prompts, the Skywork reward model favors the white or male prompt in 69.0% and 73.8% of comparisons, and the preference is larger for responses sampled under those prompts (Tab. 1). Within-prompt normalization in GRPO cancels the constant part of this preference but not the own-prompt bonus, so helpfulness-only training rewards the policy for tailoring responses to each arm. The paper proposes four objectives that connect the two arms, and the final one compares both sets of responses under the same prompt so that prompt-specific reward offsets cancel. It builds 2,582 counterfactual pairs and 725 demographic-free prompts from four public corpora. Experiments post-train Qwen3-4B-Base for 1000 steps on four folds and evaluate on DiscrimEval, Difference Awareness, BBQ, and a four-rubric LLM judge with three judge models. Transformed Reward Expectation Matching reports the highest judge scores and reduces the reward gap between the arms from +3.24 under vanilla GRPO to between +0.41 and +0.97.

Critical strengths
  1. The paper measures the reward model rather than assuming it is biased. It scores the same response under both prompts, so any difference comes from the prompt alone (App. A). This identifies the mechanism behind the paper's central claim and does not depend on the LLM judge.

  2. The final objective rests on an identification argument rather than an empirical choice. Reward models fix scores only up to a prompt-specific constant, so comparing the two arms directly confounds a difference in behavior with a difference in reward scale. Scoring both sets of responses under the same prompt removes that constant.

Critical weaknesses
  1. The comparison basis is too narrow to support the claims. All experiments use one policy model and one reward model. The amplification mechanism is measured as a property of Skywork (App. A), so neither the amplification nor the effectiveness of the mitigations is established for any other reward model. The only training baseline is vanilla GRPO. Section 2 cites prior mitigation methods, including BiasDPO (Allam, 2024), BiasGRPO (Reddy et al., 2026), consistency training (Irpan et al., 2025), and causal reward models (Wang et al., 2025b), and none is evaluated. Response Matching resembles the consistency-training family, but it is the paper's own construction rather than a published method run under its reported settings. The results therefore establish improvement over unmitigated training in a single configuration.

  2. The headline improvement rests on a measure that cannot be checked. The four-rubric LLM judge carries the main claim, and the rubrics and class labels will not be released (Sec. 5 and the Ethics Statement). Inter-judge agreement is fair, with mean pairwise quadratic-weighted κ\kappa between 0.260 and 0.386 (Tab. 6). No numerical results or measures of variability are reported anywhere, only training curves. The public benchmarks do not corroborate the claim. BBQ does not separate the methods (App. I.1), and the DiscrimEval gap grows under every objective and ends far above the initializer (Sec. 5). Difference Awareness is the only public benchmark that separates the methods.

Questions for authors

Clarification requests:

  1. Report a table of numerical results for all methods and benchmarks at a stated checkpoint, with variability across folds.
  2. State whether the rubrics and the class labels will be released. The main claim rests on them and cannot be verified otherwise.
  3. State what the agreement in Tab. 6 implies for the differences between methods in Fig. 4, given that κw\kappa_w lies between 0.260 and 0.386.
  4. Reconcile the claim of improved invariance with BBQ not separating the methods and with the DiscrimEval gap growing under every objective.
  5. Transformed Reward Expectation Matching answers fewer difference-required Difference Awareness items than vanilla GRPO and Reward Symmetrization under mixed training (App. H.2 and H.5). State the trade-off this implies between invariance and appropriate differentiation.
  6. All invariance measures here are behavioral, while App. A shows the reward model's preference persists regardless of the policy's output. Prior work reports fair surface behavior alongside retained internal bias [Himelstein et al., Silenced Biases: The Dark Side LLMs Learned to Refuse, AAAI 2026]. Can the proposed objectives distinguish an invariant policy from one that suppresses the surface expression of that preference?

Requested experiments:

  1. Report the main comparison with at least one additional policy model and one additional reward model.
  2. Report the measurement of App. A for those additional reward models, since the own-prompt bonus is what drives the amplification.
  3. Compare against the prior mitigation methods cited in Sec. 2, such as BiasDPO (Allam, 2024), BiasGRPO (Reddy et al., 2026), and consistency training (Irpan et al., 2025), or state why they are not comparab
9 comments

We thank the reviewer for the detailed comments. We tried to address the concern by adding new additional experiments as requested. Here are the answers for the questions:

Second policy and reward model. We retrain vanilla GRPO and TREM with the same setup, once with Phi-4-mini-instruct as the policy and once with AllenAI Llama-3.1-8B-Instruct-RM-RB2 as the reward model. With Phi-4-mini, TREM lowers the DiscrimEval gap and raises Difference Awareness and the judge score at every α\alpha (Appendix K.3, Table 10). With RB2, TREM lowers the DiscrimEval gap and raises the judge score at every α\alpha (Appendix K.2, Figure 18). In Appendix A, Table 3, we also show that this RB2 reward model exhibits a preference for white/male prompts (where the contents are otherwise identical) to receive higher rewards.

Baselines. We added BiasGRPO, which trains with GRPO on a reward model for unbiased responses (Appendix K.1). Its reward alone collapses helpfulness, and mixed with the Skywork reward it does not lower the DiscrimEval gap. We did not run the other methods. BiasDPO needs a biased and an unbiased completion per prompt, which our open-ended prompts lack. Consistency training is an SFT method, and our SFT initializer already shows low bias, while most of the bias appears during RL post-training (Figure 3). Causal reward models retrain the reward model, which our setting avoids, since we keep the existing reward model fixed.

Numerical results. We used curves because our claim is about how bias grows during training, which a single checkpoint hides. We added results for all methods with mean ± standard deviation across the folds in Tables 1, 2, and 10.

Release of rubrics and labels. We release the counterfactual prompt pairs, but not the class labels, the annotation policy or the rubrics (the rubrics and policies contain labeled few-shot examples which imply the authors’ choices on when demographics are relevant). Deciding when demographics should change an answer is a normative choice, and a released label set would likely become a standard that later work treats as ground truth, which is not the contribution we want this paper to be remembered for. Users can supply their own labels and run the same method. Appendix I.3 supports this. On agnostic pairs, where either label is reasonable, our mitigations score higher than vanilla GRPO under both rubrics.

However, for starting inspiration, the content of our rubrics can be approximated by the names of the judge axes: substance invariance, level capabilities, tonal neutrality, no unsolicited commentary, judiciousness, tonal appropriateness, and utilization.

Agreement Scores of Judges. The κ\kappa values lie between 0.260 and 0.386 because the judges often give the top score of 4 out of 4. Since two judges are likely to both give a 4 by chance, κ\kappa stays small even when they agree. On the other hand, all three judges give exactly the same score on 65% to 81% of ratings. This is why we report the median score of the three judges in all plots and tables, which reduces the noise from any single judge. Each point in Figure 4 also averages many ratings, so a few disagreements do not change the overall results.

Reconciling the invariance claim. We claim that the objectives reduce the bias that RL adds, not that they remove it, and Section 5 already states that no objective returns the model to the invariance of its initializer. Under invariant-only training, vanilla GRPO ends with the largest DiscrimEval gap and every objective lowers it (Table 1). Under mixed training, all methods end at the same gap, as Appendix I.1 reports. BBQ does not separate the methods because every run, including vanilla GRPO, stays below the initializer’s ambiguous bias, while vanilla GRPO lowers all four open-ended rubric scores. Forced-choice benchmarks miss these failures, which is why we propose an open-ended evaluation method.

Mixed training trade-off. We agree that this is a trade-off. In mixed training, TREM changes the reward only on invariant pairs, and dependent pairs receive the standard reward. Learning to ignore demographics on invariant prompts likely makes the model somewhat less willing to treat groups differently, which lowers its accuracy on Difference Awareness questions that require different treatment. In open-ended answers to dependent prompts, this effect does not appear. TREM’s dependent judge score stays near the initializer while vanilla GRPO’s drops, and TREM’s utilization score stays similar to the other methods and above the initializer (Appendix I.2).

Invariance versus suppression. Himelstein et al. study safety-aligned models whose refusals hide biases. In our setting, we always start from a base model without safety alignment and train only for helpfulness, so our models do not learn to refuse. Beyond the sampled text, DiscrimEval evaluates the model on its log-probabilities, so it measures the model’s underlying preference and not only its wording. We do not aim to remove demographic information from the model. We want the model to use it only when the task requires it, as in our dependent prompts.

Reviewer_BSuKOpenReview ↗

Response to rebuttal

Thank you for the response and for the additional experiments. The second policy and reward model address the generalization concern, and the numerical tables address the missing results. The comparison basis remains thin, as only one prior method was evaluated while three others were declined despite being feasible. A second concern is the decision not to release the labels and rubric. They set both what is trained on and how it is scored, and neither can be examined. The stated reason is that publishing such a judgment would turn it into a de facto standard, yet releasing it is the standard in the field. BBQ, DiscrimEval and Difference Awareness release the labels encoding comparable judgments, and First-Person Fairness publishes the rating prompt of its LLM judge, all four being works this paper cites. My score therefore remains unchanged.

Reviewer_BqWXOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In general, prompts with the same semantic meaning should give similar responses. However, this is often not the case for LMs; in particular, LMs can give very different responses to prompts that differ only on protected characteristics (e.g. gender, race). The authors develop a reinforcement learning objective (and training procedure involving generating counterfactual prompts and responses) that alters that advantage calculation in GRPO that penalizes models for giving materially different responses for semantically similar prompts.

Critical strengths
  • The sensitivity of responses to minor variations in prompt content or to protected characteristics is an important problem, and this is a clever solution that seems to work reasonably well.
  • The counterfactual editing of the responses is quite clever.
Critical weaknesses
  • My copy of the paper seems to be missing the Appendices. I'm particularly interested in seeing sample generations, since reward models are weird and I could imagine that models make responses score more similarly by making both of them strange in some way. The authors mention some qualitative examples of this in Section 4.1, but I'd be curious to see actual generations. Matching rewards may also not give semantically similar generations.
  • I'd also be curious to see what the transformed responses look like — it seems plausible that a reward model would score a grammatically strange response poorly, so this might harm the rewritten responses. If certain kinds of responses are more harmed by this than others, this could introduce unintended bias.
  • Similarly, GRPO can be unstable so I was curious about some of the training details.

Clarity

  • I got a bit confused in the "Reward Expectation Matching" section since I thought this was going to be the "Transformed Expectation Matching" setup. It might be helpful to include a table with clarifying examples showing the differences between the multiple setups.
Questions for authors
  • See above for questions in the weaknesses section
  • It's not clear to me that the within group normalization of GRPO is helpful here. Is there a reason to use GRPO over any other policy gradient method?
  • It seems desirable that models would give comparable answers for semantically equivalent prompts, but they don't. Any other domains or areas that you tried that worked (or didn't work)? Do you expect this to generalize to other settings?
3 comments

We thank the reviewer for their positive feedback. We added the appendices as a separate file under the supplementary material as a PDF. It can be found under the abstract on top of this page.

Generations. Appendix L shows sample generations with both responses of a pair side by side (Tables 11 to 13). For illustration, in one example, the prompt asks for an assignment that assesses a male or a female student's ability to visualize political data. Vanilla GRPO explains that the assignment suits the male student because it aligns with STEM and tech career readiness, and that it suits the female student because it lets her show creativity and follows gender-equitable pedagogy. TREM writes the same assignment for both students without such framing, and the paired judge scores its pair 100.00 against 85.94 for vanilla GRPO (Table 13).

Transformed Responses. We use deterministic word swaps for transforming the sentences. For example, it maps “I gave her the report, and her manager approved it” to “I gave him the report, and his manager approved it”. We also note that this method can still mishandle rare pronoun constructions and also swap words that refer to people other than the subject itself. However, we preferred this simple approach instead of increasing the computational complexity of the overall approach. Appendix B and Table 4 give the swap rules and examples.

GRPO training details. We sample 16 responses per prompt at temperature 1.0 and train with Adafactor at a learning rate of 5×10−65 \times 10^{-6}, a KL coefficient of 10−310^{-3}, a clipping coefficient of 0.2, gradient clipping at 1.0 and 30 warmup updates, for 1000 steps. We sweep α∈{0.05,0.10,0.20}\alpha \in \{0.05, 0.10, 0.20\} for TREM. Appendix G.2 and Table 8 list all settings, and Appendix G.3 reports the α\alpha sweep.

Naming in the Reward Expectation Matching section. Thank you for catching this. We now name it transformed reward expectation matching in the introduction. We also add Table 5 in Appendix C, which compares the four objectives and gives an example of each.

Why GRPO over other policy gradient methods? We use GRPO because it is one of the most widely used methods for RL post-training of LLMs, and we study the bias that this standard recipe introduces.

Other domains and settings. We study black/white race and female/male gender. The objectives need only a counterfactual pair, a relation label and a transformation τ\tau, so they apply in principle to other demographic variables such as age, religion or nationality. We have not tested these settings and list them as future work.

Thanks for the response. Apologies for the oversight with the appendix; this answers most of my questions. While I generally prefer erring on the side of transparency, I'm much more willing to give the benefit of the doubt on full release of labels than the other reviewers. I'm also more willing to give the benefit of the doubt on some of the fiddly details (LLM as a judge, etc.) since the method seems quite clever, and operationalizing things enough to run experiments requires making some decisions.

For other domains, I think something like this procedure could be quite valuable for optimizing for self-consistency (Pres 2026*) beyond just mitigating bias. Potentially an interesting direction if the authors wish to extend this work.

*https://time-for-consistency.github.io/

Thank you for the follow-up and for your support of the work. We appreciate the pointer to self-consistency work. We will definitely add this as a future work as a natural extension and we also think that it would be useful to combine it with counterfactual invariance. Additionally, as detailed in the general response, we also decided to release the rubrics and a labeled subset of the pairs.

Reviewer_Ch7GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper basically aims to prevent large language models from treating users different based on their demographics when that information is irrelevant to the current task.

To do this, the authors basically train with counterfactual pairs so that rollouts are (mostly) matching.

It is then evaluated using proxy metics and LLMs as a judge.

Critical strengths
  • One of the paper's core strengths is simply focusing on an important problem, there is plenty of evidence that current post training methods are not really aligning LLMs the way we want.
  • Evaluation generally favours the method compared to GRPO etc.
Critical weaknesses
  • The LLM as a judge is not directly comparing counterfactual rollouts, instead they see one at a time and are asked to "imagine" if a counterfactual would be appropriate. This contradicts the paper's main objective and "hand waves" over directly evaluating it.
  • The baselines are not great, improvement over vanilla GRPO are unsurprising when explicitly optimising for counterfactual invariance, which GRPO does not do.
  • The method relies on defining the invariance, which strikes me as non-trivial
  • The authors explicitly decline to release classification labels, the annotation policy, evaluation rubrics and dataset splits, preventing faithful reproduction
  • Experiments rely on a limited number of LLMs a wider net would be more convincing.
Questions for authors

See weaknesses

2 comments

We thank the reviewer for their valuable comments. Here, we address the concerns:

The judge does not compare rollouts directly. We added a paired-response judge that reads both responses of a pair side by side, in both orders, with the same judges and rubric (Section 4.1, Table 2). It ranks the methods in the same order as the single-response judge. TREM lowers the gap between arms from 13.7 to 5.6 points and the share of pairs judged different from 46.4% to 31.8%. The paper also compares the two arms directly without a judge. When the reward model scores the responses of both arms under the same prompt, the gap between their mean rewards ranges from +0.03 to +0.71 at initialization, +1.44 to +3.24 after vanilla GRPO, and +0.41 to +0.97 after TREM (Figure 5).

Baselines. We use vanilla GRPO to show that standard RL amplifies counterfactual bias, and we compare our mitigation methods to it. Since counterfactual invariance under a fixed reward model is not a well-explored research direction, we focus the paper on defining and showing the problem, and then propose a mitigation that keeps the reward model fixed, so it applies to any reward model trained with a Bradley-Terry objective. However, we added BiasGRPO as a baseline, which trains with GRPO on a reward model for unbiased responses (Appendix K.1). Its reward alone collapses helpfulness, and mixed with the Skywork reward it does not lower the DiscrimEval gap. We hypothesize that a general bias reward does not target counterfactual invariance.

Defining invariance is non-trivial. We agree that defining invariance is not easy. However, training a language model already requires policies for its safety, so developers already need a clear definition of bias. We developed our policy with domain experts. Our method takes this policy as one label per counterfactual pair, much like the classes of a classifier. A different policy changes which pairs are labeled invariant, not the method itself.

Labels, policy, rubrics and splits are not released. We release the counterfactual prompt pairs, but not the class labels, the annotation policy or the rubrics (the rubrics and policies contain labeled few-shot examples which imply the authors’ choices on when demographics are relevant). Deciding when demographics should change an answer is a normative choice, and a released label set would likely become a standard that later work treats as ground truth, which is not the contribution we want this paper to be remembered for. As the previous bullet point explains, users can supply their own labels and run the same method. This is demonstrated by the results of the agnostic label ablation in Appendix I.3: scoring the same outputs under both the dependent and invariant rubrics, our mitigation improves or matches the baseline on every axis of both rubric sets.

However, for starting inspiration, the content of our rubrics can be approximated by the names of the judge axes: substance invariance, level capabilities, tonal neutrality, no unsolicited commentary, judiciousness, tonal appropriateness, and utilization.

A limited number of LLMs. We retrain vanilla GRPO and TREM with Phi-4-mini-instruct as a second policy and AllenAI Llama-3.1-8B-Instruct-RM-RB2 as a second reward model. TREM lowers the DiscrimEval gap and raises the judge score with both (Appendix K.3, Table 10, Appendix K.2, Figure 18).

AuthorsOpenReview ↗

Dataset Release and Reminder

Following the discussion, we will release the counterfactual prompt pairs, the judge rubrics, and a labeled subset of the pairs with the camera-ready version, and share the full label set on request under a research-only license, as detailed in the general response. We believe that this addresses the reproducibility concern. We hope these changes address your concerns and would appreciate it if you could reconsider your assessment.

Since the discussion period ends soon, we would be grateful if you could let us know whether any concerns remain and reconsider your assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their feedback and for unanimously acknowledging the problem’s importance. The main concerns were around the generalization to additional models, evaluation procedure, numerical result formatting, and the decision behind unreleased labels and rubrics. We address them in this rebuttal.

Scope of the paper. Our aim is not a new RL algorithm. Our work aims to contribute two findings, scoped to enhance utility and practical applicability: 1) the inadequacy of classical QA-based fairness benchmarks to capture the real-life bias of open-ended responses most commonly felt by users, and 2) mitigation as a minimal intervention to the most commonly used post-training methods and existing helpfulness reward models. Within this scope, we find that standard helpfulness-reward based RL amplifies counterfactual bias, and we trace this in part to the reward model. We then show that without retraining a reward model, we can achieve fairer responses with a small change in training reward while preserving the helpfulness gain.

We added new results, here is the short summary:

Additional Policy/Reward Model. We retrain vanilla GRPO and TREM with the same setup, once with Phi-4-mini-instruct as the policy and once with AllenAI Llama-3.1-8B-Instruct-RM-RB2 as the reward model. With Phi-4-mini, TREM lowers the DiscrimEval gap and raises Difference Awareness and the invariant judge score at every α\alpha (Appendix K.3, Table 10). With RB2, TREM lowers the DiscrimEval gap and raises the invariant judge score at every α\alpha (Appendix K.2, Figure 18). Hence, our approach transfers to other models, and vanilla GRPO amplifies the bias there as well.

Additional baseline. We add BiasGRPO [1], which trains with GRPO on a reward model for unbiased responses, so it fits the same training scheme as our methods. We train with its reward alone and with weighted sums of its reward and the Skywork reward. Its reward alone keeps the DiscrimEval gap near zero but collapses helpfulness. The 80/20 Skywork-BiasGRPO mix keeps helpfulness but ends with a DiscrimEval gap as large as that of vanilla GRPO (Appendix K.1, Figure 17).

Paired-response judge. We add a second judge that reads both responses of a counterfactual pair side by side, in both orders, with the same judges and rubric. It ranks the methods in the same order as our single-response judge (Section 4.1, Table 2).

Numerical tables. We add tables that report every method at step 1000 as means and standard deviations over the four held-out folds (Tables 1, 2, and 10).

[1] Reddy, Saket, Ke Yang, and ChengXiang Zhai. "BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

AuthorsOpenReview ↗

Update on releasing labels and rubrics

Following the discussion with the reviewers, we decided to release the following with the camera-ready version.

  • The counterfactual prompt pairs.
  • The judge rubrics.
  • The invariant, dependent, and agnostic labels for a subset of the pairs.

Because of ethical concerns about establishing normative judgments, we will make the full label set available on request under a research-only license that excludes its use as a benchmark or as ground truth. We will update Section 5 and the Ethics Statement to match. We hope this resolves the reviewers’ concerns about reproducibility and transparency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.