- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper asks whether weight-level bias mitigation that removes behavioral bias also removes bias from a model's internal representations. The authors fine-tune Mistral-7B and four Qwen2.5 models to say yes to White-named and no to Black-named resumes, locate the injected bias direction with a double-contrast estimator, then apply ablation, LoRA and OFT and measure residual behavioral bias, residual activation-space bias, and balanced accuracy.
Critical strengths
The question matters: whether interventions on LLM weights leave the demographic signal recoverable inside the model is open and consequential. The double-contrast estimator, which subtracts the baseline model's within-pair difference, is a sensible control, and the paired-resume design with placebo name pairs over fifteen seeds shows care. The authors are also candid about several of their limitations.
Critical weaknesses
Unfortunately the setup cannot answer the question it poses. Injection collapses every model to balanced accuracy 0.50 with behavioral bias 1.000, so the "biased" model is a name classifier, not a hiring model with a skew. LoRA and OFT are then trained on the task labels and the bias disappears because the model learns the task, while ablation, as Section 4.2 states, receives no corrective training signal at all. Concluding that ablation "fails" from this comparison is not a finding about ablation. Worse, the Qwen2.5 baselines sit at 0.51-0.56 balanced accuracy before any injection, including at 14B, which points to a broken task or prompt rather than a model limitation, and leaves Mistral-7B (0.74) as the only informative model; the cross-family and cross-scale claims do not stand on that.
The activation-space metric also cannot detect the failure the paper is looking for. It measures residual alignment with one direction, at one layer, fitted before mitigation; if mitigation rotates the signal elsewhere, the metric reports success, which is exactly the "lipstick on a pig" failure the paper cites. The remedy is in the literature the paper already cites (INLP, LEACE, RLACE): train a probe on post-mitigation activations and check whether race is still recoverable. The paper cites all three and runs none.
Questions for authors
- What happens with ablation followed by the same benign fine-tuning that LoRA and OFT receive?
- Can a probe trained on post-mitigation activations still recover race?
- Why is the Qwen2.5 family at chance on the task at every scale, including 14B?
Response to Reviewer XDai Comments
We thank Reviewer XDai for their insightful comments. Below, we endeavor to address the questions raised.
On the activation-space metric not detecting the failure the paper is looking for. We want to clarify that our activation-space metrics do not rely on alignment alone. Our cosine similarity metric measures orientation relative to the identified bias direction, and we clearly noted that this metric would not be sufficient. This is why we introduce our presence and retention fraction metrics, which measure the magnitude of the residual bias vector. This allows us to fully measure the effect of the bias removal rather than relying on rotation. Q2 addresses the point on a probe being trained on post-mitigation activations. For more details, please see Appendix C for more details.
We have also run experiments with LEACE and INLP. Both reduce a linear probe to chance (0.500 and 0.475 on Qwen2.5-7B). RLACE optimizes the same bound LEACE attains in closed form for a binary concept; therefore, the LEACE measurement reports are the same for both methods.
Q1: We appreciate the reviewer pointing out that the ablation and LoRA/OFT comparison was unmatched because of the training signal. We have tested the proposed question and received the following results:
We see that adding a training signal allowed it to perform better than the untrained method. Given this, we will remove the claim that ablation "fails" compared to LoRA and OFT. However, we want to point out that the appropriate revised claim would be that ablation is redundant and contributes a neglible effect on top of benign fine-tuning. By doing so, we ensure a matched comparison that still supports our original claim of ablation not being as effective as the other methods for our study.
Q2: We tested this question on Mistral-7B and found the following results:
We see that despite LoRA and OFT bringing demographic disparity close to 0.00, race was still recoverable. We acknowledge that OFT's lower balanced accuracy may have affected probe accuracy. Regardless, these results support the core thesis of our paper.
On the Qwen2.5 baselines and Q3: Please see the general note that will soon be posted to all reviewers.