acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Weight-Level Bias Mitigation Methods Against Behavioral and Activation-Space Bias

Abstract

Ensuring language models are free of harmful bias matters wherever their outputs shape real decisions. Yet, bias evaluation has relied almost entirely on behavioral metrics, leaving open whether mitigation methods actually remove bias from a model's internal representations. Fewer studies still examine whether weight-level methods, techniques that directly manipulate a model's weights, can eliminate both behavioral and activation-space bias simultaneously. We test whether ablation, low-rank adaptation (LoRA), and orthogonal fine-tuning (OFT) can remove injected behavioral and activation-space bias across different LLM families and model sizes. We devise a novel double-contrast, mean-of-differences bias identification technique that selects the layer where the bias signal in the model's representations is strongest and most consistent with the contrasting demographics. Our experiments show that injected bias becomes more concentrated as layer depth increases, but this correlation does not establish which layers cause the bias. Ablation fails to remove behavioral bias and does not restore the model's task capability, even when constructed explicitly to remove the bias direction. LoRA and OFT remove behavioral bias in all cases, but their effect on activation-space bias varies across models. These findings show that evaluating behavioral or activation-space bias alone is insufficient, motivating the need to assess both simultaneously.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Bias mitigation, bias identification, behavioral bias, activation-space bias, LLM bias

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XDaiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks whether weight-level bias mitigation that removes behavioral bias also removes bias from a model's internal representations. The authors fine-tune Mistral-7B and four Qwen2.5 models to say yes to White-named and no to Black-named resumes, locate the injected bias direction with a double-contrast estimator, then apply ablation, LoRA and OFT and measure residual behavioral bias, residual activation-space bias, and balanced accuracy.

Critical strengths

The question matters: whether interventions on LLM weights leave the demographic signal recoverable inside the model is open and consequential. The double-contrast estimator, which subtracts the baseline model's within-pair difference, is a sensible control, and the paired-resume design with placebo name pairs over fifteen seeds shows care. The authors are also candid about several of their limitations.

Critical weaknesses

Unfortunately the setup cannot answer the question it poses. Injection collapses every model to balanced accuracy 0.50 with behavioral bias 1.000, so the "biased" model is a name classifier, not a hiring model with a skew. LoRA and OFT are then trained on the task labels and the bias disappears because the model learns the task, while ablation, as Section 4.2 states, receives no corrective training signal at all. Concluding that ablation "fails" from this comparison is not a finding about ablation. Worse, the Qwen2.5 baselines sit at 0.51-0.56 balanced accuracy before any injection, including at 14B, which points to a broken task or prompt rather than a model limitation, and leaves Mistral-7B (0.74) as the only informative model; the cross-family and cross-scale claims do not stand on that.

The activation-space metric also cannot detect the failure the paper is looking for. It measures residual alignment with one direction, at one layer, fitted before mitigation; if mitigation rotates the signal elsewhere, the metric reports success, which is exactly the "lipstick on a pig" failure the paper cites. The remedy is in the literature the paper already cites (INLP, LEACE, RLACE): train a probe on post-mitigation activations and check whether race is still recoverable. The paper cites all three and runs none.

Questions for authors
  1. What happens with ablation followed by the same benign fine-tuning that LoRA and OFT receive?
  2. Can a probe trained on post-mitigation activations still recover race?
  3. Why is the Qwen2.5 family at chance on the task at every scale, including 14B?
1 comment
AuthorsOpenReview ↗

Response to Reviewer XDai Comments

We thank Reviewer XDai for their insightful comments. Below, we endeavor to address the questions raised.

On the activation-space metric not detecting the failure the paper is looking for. We want to clarify that our activation-space metrics do not rely on alignment alone. Our cosine similarity metric measures orientation relative to the identified bias direction, and we clearly noted that this metric would not be sufficient. This is why we introduce our presence and retention fraction metrics, which measure the magnitude of the residual bias vector. This allows us to fully measure the effect of the bias removal rather than relying on rotation. Q2 addresses the point on a probe being trained on post-mitigation activations. For more details, please see Appendix C for more details.

We have also run experiments with LEACE and INLP. Both reduce a linear probe to chance (0.500 and 0.475 on Qwen2.5-7B). RLACE optimizes the same bound LEACE attains in closed form for a binary concept; therefore, the LEACE measurement reports are the same for both methods.

Q1: We appreciate the reviewer pointing out that the ablation and LoRA/OFT comparison was unmatched because of the training signal. We have tested the proposed question and received the following results:

ConditionDemographic disparityBalanced accuracy
Injected model1.0000.500
Ablation only0.1600.470
Benign fine-tuning only0.0010.855
Ablation, then benign fine-tuning0.0040.815

We see that adding a training signal allowed it to perform better than the untrained method. Given this, we will remove the claim that ablation "fails" compared to LoRA and OFT. However, we want to point out that the appropriate revised claim would be that ablation is redundant and contributes a neglible effect on top of benign fine-tuning. By doing so, we ensure a matched comparison that still supports our original claim of ablation not being as effective as the other methods for our study.

Q2: We tested this question on Mistral-7B and found the following results:

Mistral-7BDemographic disparityBalanced accuracyProbe accuracy
Baseline model——0.992
Injected model1.0000.5001.000
+ LoRA0.0000.9100.850
+ OFT0.0110.6800.877

We see that despite LoRA and OFT bringing demographic disparity close to 0.00, race was still recoverable. We acknowledge that OFT's lower balanced accuracy may have affected probe accuracy. Regardless, these results support the core thesis of our paper.

On the Qwen2.5 baselines and Q3: Please see the general note that will soon be posted to all reviewers.

Reviewer_MpyEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the effect of weight-level debiasing methods on both behavioral bias and activation-space bias removal. Standard setups with SFT style post-training for bias addition and removal are studied with mostly empirical findings.

Critical strengths
  1. Mostly well-written is easy to follow.
  2. Interesting setup and main research direction in understanding residual / implicit and hard to measure biases in LLM behavior from the inside and outside simultaneously.
Critical weaknesses
  1. Limited novelty is my main concern. The techniques proposed by the authors are fairly standard at this point. The difference-in-differences estimator is classic in econometrics and is already used as a standard in contrastive activation addition (CAA) steering methods popularized by Rimsky et. al.. Joint study of behavioural and activation biases is also well known as mentioned by the authors in the related works section (Elazar et. al. and Gonen & Goldberg). Simply applying orthogonal fine tuning for bias mitigation with mostly empirical findings in of itself is not novel enough for ICLR imo.
  2. There are some major experimental concerns that confound the findings. I am not sure what the authors do is de-biasing or rather just re-training. Proper de-biasing techniques are not used and only mentioned in passing.
  3. Injecting synthetic bias via SFT and I assume LoRA and then mitigating it with another LoRA doesn't do justice to subtle biases in LLMs and their estimation. I am interested in what the authors have to say about the biases that could have existed in the models before any SFT. Is the rank of the de-biaser probe consistent with the SFT probe?
  4. Some of the findings are trivial e.g. late stage spikes are obvious if the target is the answer token.
Questions for authors

See above

2 comments
AuthorsOpenReview ↗

Response to Reviewer MpyE Comments

We thank Reviewer MpyE for their insightful comments. Below, we endeavor to address the questions raised.

W1: We respectfully push back on the concerns regarding limited novelty.

  • The double-contrast estimator is not a CAA variant. CAA (Rimsky et al.) constructs a steering vector from a single contrast and uses it to intervene. Unlike CAA, our estimator nests another contrast to isolate the injected bias from the generic activation shift introduced by fine-tuning. It accomplishes this by subtracting the baseline model's demographic difference from the biased model's difference. A single contrast cannot do this because it captures the fine-tuning shift along with the demographic signal, entangling the two and making it insufficient for our problem. For more details, please see Section 4.1.
  • Our evaluation does something prior joint studies cannot. Elazar et al. and Gonen & Goldberg established their claims on static word embeddings and naturally occurring bias with no ground truth. Unlike these works, we evaluate on modern LLMs against a controlled, injected ground truth. Specifically, we injected ground truth to address the limitations Barrett et al. raised about the Elazar et al. study, where a lack of robustly encoded bias led to poor generalizability. For more details, please see Section 2.

W2: We appreciate the reviewer's concerns about re-training. We are currently running an experiment comparing our results with what happens if we fine-tune the biased model on a neutral objective with no counter-bias signal. If additional fine-tuning were responsible for removing the bias, then the model fine-tuned on a neutral objective would become debiased.

We mentioned proper de-biasing techniques such as INLP and LEACE, but we want to clarify that we focused on weight-level mitigations. INLP and LEACE are inference-level mitigations.

W3: We considered biases existing in the models before SFT, and we do measure them. In the baseline models, a linear probe recovers the applicant's race at 0.992 (Mistral-7B) and 0.994 (Qwen2.5-7B), while the same models show essentially no behavioral disparity (−0.007 on Mistral). The representation encodes the attribute even where the behavior does not use it. This is why our estimator uses a double contrast. We subtract the baseline model's own within-pair difference, isolating what the injection added from the demographic structure already present. Candidly, naturally occurring bias is hard to pinpoint without a ground truth, which is why we injected it. Doing so amplifies real-world bias to a magnitude we can attribute and measure.

Regarding rank consistency, the injection uses r=32r=32, the de-biasing LoRA r=16r=16, and OFT r=8r=8, with identical target modules. Both de-biasers reduce demographic disparity to near zero while preserving task accuracy, so the lower rank was not a limiting factor. We note that OFT's rr denotes the number of orthogonal blocks rather than a low-rank factorization dimension, so it is not directly comparable to LoRA's rank.

W4: We endeavored to report all relevant findings to ensure a thorough analysis. However, we are open to moving more trivial results to the Appendix. We will note that, alongside the late-stage spike finding, bias is not causally localizable. Hence, we added that finding to further contextualize this more novel finding in the absence of causal localizability.

AuthorsOpenReview ↗

Update on Response to Reviewer MpyE Comments

This updates our previous post and includes results from our experiment comparing a model that receives a neutral fine-tuning signal with one that receives a counter-bias signal. We hope these results will clarify the rest of Reviewer MpyE's queries.

To test whether further fine-tuning removes the bias, we ran a matched experiment that differed only in whether it received a counter-bias signal or a neutral fine-tuning signal. The counter-bias arm uses merit-labeled resumes with names present, and labels decorrelated from names. The neutral arm uses identical resumes and labels with applicant names stripped from the training text. Names are restored at evaluation for both.

Below are the results. The top table shows behavioral bias metrics, and the bottom shows activation-space bias metrics.

Test conditionQwen2.5-7B Demographic disparity ΔQwen2.5-7B Balanced accuracyMistral-7B Demographic disparity ΔMistral-7B Balanced accuracy
Baseline model B0.0110.5600.0040.740
Injected model M_b1.0000.5001.0000.500
Counter-bias fine-tuning0.0040.9050.0130.895
Neutral fine-tuning0.0430.8800.0740.670
Test conditionQwen presenceQwen retentionMistral presenceMistral retention
Baseline model B0.0180.0000.4570.000
Injected model M_b0.6801.0000.9441.000
Counter-bias fine-tuning0.006−0.0190.6880.474
Neutral fine-tuning0.013−0.0080.6620.420

Both conditions close most of the behavioral bias gap on both models and the activation-space bias gap on the Qwen models. Neither condition closes the activation-space bias gap to the baseline on the Mistral models. We see that the counter-bias signal further reduces the gap, but most of the reduction occurs without it. We therefore agree with the reviewer, and we withdraw any claim that the behavioral result demonstrates targeted de-biasing.

However, this does not affect our central claim, which concerns what a behavioral audit can detect rather than how a given procedure works. The neutral test condition is itself a weight-level intervention that nearly eliminates demographic disparity, but it was unable to reduce activation-space bias to the baseline in the Mistral model. Whether the procedure is called de-biasing or re-training, a behavior-only audit would report near-zero disparity on both models, yet only on Qwen has the injected direction been removed. Following these experiments, we will add these neutral test-condition results as a standing control in the main results of our paper.

Reviewer_5Tt5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper injected LLMs with a name-based hiring bias, then showed that training-based fixes (LoRA/OFT) make models behave fairly everywhere, but only sometimes erase the bias from the model's internals, while the ablation method fails to even make the behavior fair. So, fairness must be audited both in outputs and in representations.

Critical strengths
  • The topic is timely. It is important to have models that are fair in both their outputs and representations, especially now that bias audits are becoming a legal requirement.

  • The pipeline and the manuscript are easy to follow. The authors are honest and clear about the limitations of their method.

Critical weaknesses
  • The setup measures a shortcut, not general bias. The resumes differ only in the first name, so the model can answer by simply learning a rule from the name (the bias gap reaches 1.0, and task accuracy drops to chance). The linear estimator is fine for this specific injected bias, but the broader claims (for example, for auditors) need either a test for non-linear bias or a clear statement of where the method stops applying. The small set of surnames also weakens external validity.

  • The Qwen2.5 results are confounded by baseline capability. The models were already at a random level before the bias was injected. A broken model can look debiased by accident. The main finding really depends on Mistral 7B. The paper should present Mistral 7B as the main evidence or add a model family with a stronger baseline.

  • The novelty is in the method, not in the general idea. Previous work, like https://aclanthology.org/2025.trustnlp-main.31.pdf, already reported that demographic info concentrates in later layers, and that output-level and representation-level debiasing can be disconnected. What is new here is the double contrast estimator and the causal ablation tests that show the bias cannot be pinned to one layer. The paper should state this difference more clearly.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5Tt5 Comments

We thank Reviewer 5Tt5 for their insightful comments. Below, we endeavor to address the questions raised.

On the setup measuring a shortcut, not general bias.

  • We are running an experiment with a set of held-out names to verify that the model is not simply learning a rule from the name. We will post the results in an update.
  • We address the task capability dropping to chance through additional injection-strength experiments listed in the paper. We show that across a 10x range of injection strengths, the mitigation pattern of LoRA and OFT removing bias more effectively than ablation is held constant. Therefore, the core comparison is not an artifact of the saturated model. For more details, please see Appendix H.
  • We agree that a non-linear bias test is needed to support the broader claims of the paper. We ran additional tests on the 7B versions of Qwen2.5 and Mistral models to address this suggestion. We measured the accuracy of three probes (linear, MLP, and RBF-SVM) in recovering an applicant's race after using LEACE to remove the candidate's demographic information from the model's activations.
Probe used to recover applicant raceBefore erasureAfter LEACE
Linear1.0000.500
MLP1.0001.000
RBF-SVM1.0001.000

These results show that the linear probe captures only a limited scope of bias, whereas non-linear probes recover more demographic information that the linear probe cannot capture. As a result, our revised version of the paper will explicitly state that our estimator can address only linear bias. We mention in our future work that expanding to non-linear bias is a logical next step.

  • We are running additional experiments that expand the set of surnames and will include results in an update.

On the Qwen2.5 results being confounded by baseline capability. We are happy to center the main evidence on Mistral-7B, with the Qwen2.5 models as supplementary evidence. Please also see the general note that will soon be posted to all reviewers with additional results explaining this.

On the novelty being in the method, not in the general idea. We appreciate the reviewer acknowledging the novelty in our double contrast estimator. In our revised paper, we will incorporate this suggestion and state the difference more clearly. We will also cite the paper provided in our related work.

AuthorsOpenReview ↗

Update on Response to Reviewer 5Tt5 Comments

This is an update from our previous post and includes results from our experiments on held-out names and an expanded surname bank. We hope these results will clarify the rest of Reviewer 5Tt5's queries.

Held-out names experiment: To test whether the model is learning a rule from the name rather than a demographic concept, we rebuilt the resume splits so that injection and evaluation draw from disjoint first-name banks within each group and gender. We found the following results:

ModelEvaluated onDemographic Disparity Δ
Baseline model Bseen names0.004
Baseline model Bheld-out names−0.003
Injected model M_bseen names1.000
Injected model M_bheld-out names0.998
Activation-space MetricValue
Cosine similarity (cos(v_bias seen names, v_bias held-out names))0.954
Presence on seen names0.835
Presence on held-out names0.623

The bias strongly transfers to held-out names behaviorally, but less so in the activation space. However, the internal bias direction is nearly the same whether built from seen or held-out names. **Therefore, we believe that the model is most likely not learning a rule simply from the names. We will acknowledge in our revisions that transfer between seen and held-out names is not total in the activation space.

Surname coverage: We expanded our bank of surnames to 50, drawing from the Census Bureau's frequently occurring US surnames (Comenetz, 2016) and excluding surnames with racial associations. Because surnames are held constant within each contrast pair, they carry no demographic signal. Our injected checkpoints were fine-tuned on the original ten surnames, so we tested whether the measured effect is observed on the new expanded list:

MetricMistral-7BQwen2.5-7B
Demographic disparity Δ, 10 surnames1.0001.000
Demographic disparity Δ, 50 surnames0.99991.0000
Cosine similarity (cos(v_bias 10 surnames, v_bias 50 surnames))0.9970.999
Presence on 10 surnames0.9280.672
Presence on 50 surnames0.9310.676

We see that the effect is maintained across both conditions. We agree the original ten surnames were Irish/British-skewed, and we will correct this in the revision by using the 50 surnames as reported. We note it remains a US-centric bank, which widens external validity along one axis, not all of them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Note to All Reviewers Re: Qwen2.5 Capability Results

We thank all reviewers for taking the time to provide valuable and insightful comments. This general note answers questions about the Qwen2.5 results. Specifically, it addresses Reviewer 5Tt5's concern that results may be confounded by baseline capability and Reviewer XDai's question about why the Qwen2.5 family is at chance on the task at every scale, including 14B.

The Qwen2.5 family's balanced accuracy sits near chance at every scale because it is a calibration artifact of the capability metric rather than a property of the models. We compute the capability metric as accuracy under a hard decision at P(shortlist)>0.5P(shortlist) > 0.5. However, as shown in the table below, this threshold works only if the model's probabilities are centered near 0.5. However, this does not apply to the Qwen2.5 models.

MetricMistralQwen2.5
Mean P(shortlist), all candidates0.5260.183
Mean P(shortlist), qualified0.7970.391
Mean P(shortlist), unqualified0.2870.000
Separation (qualified − unqualified)0.5110.391

Calculating separation shows how well the model distinguishes qualified from unqualified candidates (higher = better discernment). We see that Qwen2.5 has separation somewhat comparable to Mistral's, suggesting there is a merit signal. However, because both mean PP values for the Qwen2.5 models fall below the 0.5 threshold, it labels almost everyone as a "No" for shortlisting, making accuracy appear to collapse at chance.

This does not affect our main results. None of our behavioral and activation-space bias metrics rely on this threshold. Only the capability metric is affected. In our revised paper, we will report separation either alongside or in replacement of the original capability metric to ensure clarity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.