- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that pairwise bias benchmarks like StereoSet only compare a target group against one alternative, so a model can look fair on them while still favoring some groups over others. Instead, it fixes a completion (e.g., "He loved to drink Vodka."), swaps the group in the context across 5 groups, and measures how far the resulting group distribution is from uniform.
To do this the authors build Multigroup-StereoSet from StereoSet's intersentence split, adding 4 distractor groups per example chosen to be maximally spread out (Inglehart-Welzel map for nationality, Census hierarchy for occupation). They then propose DAID: first select MLP neurons whose suppression pushes the distribution toward uniform, using a first-order approximation, then fine-tune only those neurons' down-projection weights with a KL retention term.
Experiments on four models up to 3B show the max/min group probability ratio dropping from ~23-35 to ~1.3-2.3 on nationality (and from ~100 to under 1.5 on occupation), with ICAT gains of 6-16 and an MMLU drop of 1-5 points. They also show that simply ablating the selected neurons flattens the distribution but breaks the model.
Critical strengths
The multigroup formulation is the main contribution and I find it useful. Holding the completion fixed and varying the group over several dispersed groups is a stricter test than the usual stereotype vs. anti-stereotype pair. The results back this up: several baselines that improve ICAT (BiasEdit, FBA/BBA) still have max/min ratios of 15-121 on nationality (Table 11), so pairwise gains clearly don't imply multigroup parity. This is a point the community should take into account when evaluating debiasing methods.
The ablations are also more candid than usual. The zero/mean ablation in Sec. 6.1 shows that suppressing the selected neurons flattens the distribution but breaks the model, and the authors use this to argue that bias-carrying neurons are shared with general computation rather than overclaiming localization.
Critical weaknesses
-
The evaluation is essentially in-distribution. Training and evaluation use the same StereoSet contexts and completions; only the distractor groups change (dispersed vs. random). With ~650 nationality examples trained for 20-100 epochs, a near-uniform ratio on the eval split is not strong evidence of generalization. The headline numbers depend on this, so I would need to see results on held-out contexts or held-out target groups, or on an external multigroup benchmark (e.g., SoFa), before trusting them.
-
It is unclear how much the neuron discovery step actually contributes. The random-neuron control, trained the same way, already gets close while retaining more MMLU. Since K and λ are fixed, the two are not compared at matched capability cost, and there is no LoRA or full fine-tuning baseline with the same objective. The gains could therefore come mostly from the multigroup loss rather than from localization, which weakens the method contribution.
Questions for authors
Questions that would affect my score:
-
Is there any overlap in contexts/completions between the split used for neuron selection and fine-tuning and the split used for evaluation? If so, could you report MMPR on held-out contexts, held-out target groups, or an external benchmark such as SoFa? This is the main thing that would change my assessment.
-
Can you compare DAID and the random-neuron control at matched capability cost, e.g. by sweeping λ for both and plotting MMPR vs. MMLU? A LoRA or full fine-tuning baseline with the same L_bias + λL_ret objective would also help show whether the neuron selection matters beyond the loss itself.
-
How were K, λ and the number of epochs chosen per model? Table 4 suggests they were selected using ICAT/MMPR/MMLU, which are also the reported test metrics. Was there a separate validation split?
I think the multigroup perspective is a valuable direction, and I would be glad to reconsider my assessment if the authors are able to address the questions above, particularly regarding generalization to held-out data and the contribution of the neuron selection step.
3 comments
Response By Authors (2/2)
2. "… how much the neuron discovery step actually contributes …" / "compare DAID and the random-neuron control at matched capability cost … A LoRA or full fine-tuning baseline with the same objective …"
We ran a parameter-matched LoRA [2] baseline and a random-neuron control at several , all with the same objective , data and epochs:
| Qwen2.5-1.5B, nationality | trainable params | MGD () | MMPR () | MMLU | HSwag |
|---|---|---|---|---|---|
| Base | – | ||||
| DAID, | M | ||||
| Random neurons, | M | ||||
| Random neurons, | M | ||||
| Random neurons, | M | ||||
| LoRA, , all layers | M | ||||
| LoRA, , layers | M | ||||
| LoRA, , layers | M | ||||
| LoRA, , layers | M |
- Localization. Unlike LoRA, DAID aims to identify and localize parameters attributed with biased expression. The random-neuron control aims to compare this specifically, and shows the efficacy of optimizing targeted parameters. This holds at scale as well (see experiments done on presented in our response to Reviewer U1zz).
- LoRA trains well with our objective and reaches a lower in-distribution MMPR (– vs. ), which confirms that the multigroup objective is effective. However, it does not generalize as well to other social benchmarks (StereoSet (race), BBQ (race/ethnicity) as in the paper and SoFa (race)). DAID achieves a higher ICAT and a better SS score on StereoSet, a more close to uniform distribution on out-of-distribution SoFa examples as well as comparable performance on BBQ.
| Qwen2.5-1.5B, nationality | ICAT | SS | SoFa score | SoFa MGD, | SoFa MGD, all | ||
|---|---|---|---|---|---|---|---|
| Base | |||||||
| DAID | |||||||
| Random neurons, / / | – | – | – | – | – | – | – |
| LoRA, | |||||||
| LoRA, | |||||||
| LoRA, | |||||||
| LoRA, |
- Full fine-tuning was not feasible within the rebuttal period. LoRA at a matched budget is the closest comparison we could run. The LoRA results signal the possibility that full fine-tuning might not generalize as well as targeted training does. This remains an interesting experiment to conduct, and we will add this comparison in the final draft.
- Changing does not affect random control outputs a lot. Its MMPR stays at – and its MMLU at –. Its limit is therefore not the retention trade-off: randomly chosen neurons cannot express the debiasing update at any we tried, while the selected neurons reach for MMLU point.
3. "How were , and the number of epochs chosen … Was there a separate validation split?"
The hyperparameters were selected from a small grid search using the reported metrics, based on the validation split where the distractor groups were selected at random. The Table 4 presents the grid search for . Similarly, we do a manual grid search and the chosen parameters are presented in the Appendix in Table 6. The selected hyperparameters were selected to balance the tradeoff between model degeneracy and social bias performance gains.
References
[1] Marchiori Manerba, M., Stańczak, K., Guidotti, R., & Augenstein, I. (2024). Social bias probing: Fairness benchmarking for language models. EMNLP 2024.
[2] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. ICLR 2022.
Response to Authors
Thank you for the additional experiments. The held-out context/target results and the SoFa evaluation address my main concern about generalization, so I am raising my score.
Two points remain:
Contribution of neuron discovery. The parameter-matched LoRA baseline with the same objective performs comparably or better in distribution (MMPR 1.50 vs. 1.88). The OOD advantage of DAID is modest and shown for a single model and seed. This suggests the gains come largely from the multigroup objective, so I recommend framing neuron discovery as a secondary contribution. Hyperparameter selection. Please state clearly in the paper that the hyperparameters were selected on the evaluation split.
Please also include the held-out, SoFa, and LoRA results in the revision.
Response By Authors (1/2)
We thank the valuable feedback, and for recognising that "the multigroup formulation is the main contribution" and that holding the completion fixed "is a stricter test than the usual stereotype vs. anti-stereotype pair.” Below, we address the weaknesses and questions raised by the reviewer.
1. "The evaluation is essentially in-distribution …" / "could you report MMPR on held-out contexts, held-out target groups, or an external benchmark such as SoFa?"
The training split and the evaluation split share completions and differ in the distractor groups. The evaluation is performed when the distractor groups are chosen randomly. We ran suggested evaluations and would like to also thank the reviewer for the idea. We will add this experiment in the final manuscript.
Search and training use only the training portion of each split, with the paper's hyperparameters set as in the previous experiments (K=3000, λ=0.2, 20 epochs):
On unseen contexts and unseen targets, MMPR (median max/min group probability ratio) falls by 88% (90% reduction on the training examples). MGD (average multigroup KL divergence from uniform distribution) falls by 75–78% (vs. 91% in distribution). Hence, the method is able to replicate social debiasing to unseen groups.
SoFa [1] (race; 4,552 stereotypes over 223 identities, templated). We report SoFa's own score (which is the average variance of log10PPL(xg)PPL(xg+c) for g∈G) and DDS (Delta Disparity Score, average over maxi−mini of log10PPL∗), and MGD/MMPR over NG identities sampled per stereotype (also over all 223):
DAID, trained only on Multigroup-StereoSet nationality, also improves every SoFa metric. MGD by 34–44% and SoFa's own score by 8%. The effect is smaller than in distribution, as expected for different identity terms and templated probes, but consistent. The models trained on the held-out splits improve SoFa similarly.
We will add these results and also a comparison across different baseline debiasing methods in the final manuscript.