acceptodds
Under review as a conference paper at ICLR 2027

Multigroup Social Debiasing of LLMs via Neuron Discovery

Abstract

Mitigating social biases in language models is a prerequisite for their responsible deployment. Many benchmarks and debiasing methods operationalize bias as a difference between two groups, a target group and a single alternative, so a model can reduce this pairwise gap while its preferences over the remaining groups stay biased. We define social bias as the divergence from uniform of the distribution a model induces over a set of demographic groups, with the completion held fixed so that any preference is attributable to the group. To this end, we introduce , an extension of in which each target group is complemented by distractor groups selected for maximal dispersion over real-world taxonomies, while stereotypical and anti-stereotypical completions are held fixed. We then propose DAID, a two-stage method that first identifies the MLP neurons whose suppression moves the group distribution toward uniform and then fine-tunes only these neurons' parameters. Across four models on English text, DAID reduces the median group-probability ratio from - to - on *nationality* and from - to below on *occupation*, and increases ICAT by – points, at a cost of - points on . Model-editing and activation-intervention baselines raise ICAT by up to points but leave the ratio on *nationality* between and . We find that suppressing the identified neurons without training also yields near-uniform distributions but degrades language modelling capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
social bias, debiasing, neuron attribution, fairness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZwNwOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that pairwise bias benchmarks like StereoSet only compare a target group against one alternative, so a model can look fair on them while still favoring some groups over others. Instead, it fixes a completion (e.g., "He loved to drink Vodka."), swaps the group in the context across 5 groups, and measures how far the resulting group distribution is from uniform.

To do this the authors build Multigroup-StereoSet from StereoSet's intersentence split, adding 4 distractor groups per example chosen to be maximally spread out (Inglehart-Welzel map for nationality, Census hierarchy for occupation). They then propose DAID: first select MLP neurons whose suppression pushes the distribution toward uniform, using a first-order approximation, then fine-tune only those neurons' down-projection weights with a KL retention term.

Experiments on four models up to 3B show the max/min group probability ratio dropping from ~23-35 to ~1.3-2.3 on nationality (and from ~100 to under 1.5 on occupation), with ICAT gains of 6-16 and an MMLU drop of 1-5 points. They also show that simply ablating the selected neurons flattens the distribution but breaks the model.

Critical strengths

The multigroup formulation is the main contribution and I find it useful. Holding the completion fixed and varying the group over several dispersed groups is a stricter test than the usual stereotype vs. anti-stereotype pair. The results back this up: several baselines that improve ICAT (BiasEdit, FBA/BBA) still have max/min ratios of 15-121 on nationality (Table 11), so pairwise gains clearly don't imply multigroup parity. This is a point the community should take into account when evaluating debiasing methods.

The ablations are also more candid than usual. The zero/mean ablation in Sec. 6.1 shows that suppressing the selected neurons flattens the distribution but breaks the model, and the authors use this to argue that bias-carrying neurons are shared with general computation rather than overclaiming localization.

Critical weaknesses
  1. The evaluation is essentially in-distribution. Training and evaluation use the same StereoSet contexts and completions; only the distractor groups change (dispersed vs. random). With ~650 nationality examples trained for 20-100 epochs, a near-uniform ratio on the eval split is not strong evidence of generalization. The headline numbers depend on this, so I would need to see results on held-out contexts or held-out target groups, or on an external multigroup benchmark (e.g., SoFa), before trusting them.

  2. It is unclear how much the neuron discovery step actually contributes. The random-neuron control, trained the same way, already gets close while retaining more MMLU. Since K and λ are fixed, the two are not compared at matched capability cost, and there is no LoRA or full fine-tuning baseline with the same objective. The gains could therefore come mostly from the multigroup loss rather than from localization, which weakens the method contribution.

Questions for authors

Questions that would affect my score:

  1. Is there any overlap in contexts/completions between the split used for neuron selection and fine-tuning and the split used for evaluation? If so, could you report MMPR on held-out contexts, held-out target groups, or an external benchmark such as SoFa? This is the main thing that would change my assessment.

  2. Can you compare DAID and the random-neuron control at matched capability cost, e.g. by sweeping λ for both and plotting MMPR vs. MMLU? A LoRA or full fine-tuning baseline with the same L_bias + λL_ret objective would also help show whether the neuron selection matters beyond the loss itself.

  3. How were K, λ and the number of epochs chosen per model? Table 4 suggests they were selected using ICAT/MMPR/MMLU, which are also the reported test metrics. Was there a separate validation split?

I think the multigroup perspective is a valuable direction, and I would be glad to reconsider my assessment if the authors are able to address the questions above, particularly regarding generalization to held-out data and the contribution of the neuron selection step.

3 comments
AuthorsOpenReview ↗

Response By Authors (1/2)

We thank the valuable feedback, and for recognising that "the multigroup formulation is the main contribution" and that holding the completion fixed "is a stricter test than the usual stereotype vs. anti-stereotype pair.” Below, we address the weaknesses and questions raised by the reviewer.

1. "The evaluation is essentially in-distribution …" / "could you report MMPR on held-out contexts, held-out target groups, or an external benchmark such as SoFa?"

The training split and the evaluation split share completions and differ in the distractor groups. The evaluation is performed when the distractor groups are chosen randomly. We ran suggested evaluations and would like to also thank the reviewer for the idea. We will add this experiment in the final manuscript.

Search and training use only the training portion of each split, with the paper's hyperparameters set as in the previous experiments (K=3000K = 3000, λ=0.2\lambda = 0.2, 2020 epochs):

  • Held-out contexts: 518518 train / 126126 test examples.
  • Held-out target groups: six nationalities (Bangladesh, Crimea, Eritrea, Morocco, Syria, Vietnam, which account for 137137 test examples).
Multigroup-StereoSetMGD (SS), base →\rightarrow DAIDMMPR (SS), base →\rightarrow DAID
Held-out contexts: test0.510→0.1130.510 \rightarrow 0.11320.84→2.3920.84 \rightarrow 2.39
Held-out contexts: train0.527→0.0500.527 \rightarrow 0.05021.05→2.0021.05 \rightarrow 2.00
Held-out targets: test0.491→0.1220.491 \rightarrow 0.12223.76→2.7523.76 \rightarrow 2.75
Held-out targets: train0.533→0.0500.533 \rightarrow 0.05020.01→2.0020.01 \rightarrow 2.00

On unseen contexts and unseen targets, MMPR (median max⁡/min⁡\max/\min group probability ratio) falls by 88%88\% (90%90\% reduction on the training examples). MGD (average multigroup KL divergence from uniform distribution) falls by 7575–78%78\% (vs. 91%91\% in distribution). Hence, the method is able to replicate social debiasing to unseen groups.

SoFa [1] (race; 4,5524{,}552 stereotypes over 223223 identities, templated). We report SoFa's own score (which is the average variance of log⁡10PPL(xg+c)PPL(xg)\log_{10} \frac{PPL(x_g + c)}{PPL(x_g)} for g∈Gg\in G) and DDS (Delta Disparity Score, average over max⁡i−min⁡i\max_i - \min_i of log⁡10PPL∗\log_{10}PPL^*), and MGD/MMPR over NGN_G identities sampled per stereotype (also over all 223223):

SoFaSoFa score ↓\downarrowDDS ↓\downarrowMGD / MMPR, NG=5N_G = 5 ↓\downarrowMGD / MMPR, NG=10N_G = 10 ↓\downarrowMGD, all 223223 ↓\downarrow
Base0.2790.2792.892.890.5460.546 / 53.653.60.7220.722 / 291.0291.01.4481.448
DAID0.257\mathbf{0.257}2.71\mathbf{2.71}0.360\mathbf{0.360} / 19.0\mathbf{19.0}0.456\mathbf{0.456} / 79.2\mathbf{79.2}0.807\mathbf{0.807}

DAID, trained only on Multigroup-StereoSet nationality, also improves every SoFa metric. MGD by 3434–44%44\% and SoFa's own score by 8%8\%. The effect is smaller than in distribution, as expected for different identity terms and templated probes, but consistent. The models trained on the held-out splits improve SoFa similarly.

We will add these results and also a comparison across different baseline debiasing methods in the final manuscript.

AuthorsOpenReview ↗

Response By Authors (2/2)

2. "… how much the neuron discovery step actually contributes …" / "compare DAID and the random-neuron control at matched capability cost … A LoRA or full fine-tuning baseline with the same objective …"

We ran a parameter-matched LoRA [2] baseline and a random-neuron control at several λ\lambda, all with the same objective Lbias+λLret\mathcal L_{\text{bias}} + \lambda \mathcal L_{\text{ret}}, data and epochs:

Qwen2.5-1.5B, nationalitytrainable paramsMGD (SS) ↓\downarrowMMPR (SS) ↓\downarrowMMLU ↑\uparrowHSwag ↑\uparrow
Base–0.550.5523.0423.0459.2959.2963.5963.59
DAID, λ=0.2\lambda = 0.24.614.61M0.040.041.881.8858.2858.2861.9061.90
Random neurons, λ=0.2\lambda = 0.24.614.61M0.2080.2084.884.8859.2159.2163.1263.12
Random neurons, λ=0.1\lambda = 0.14.614.61M0.2080.2084.774.7759.4559.4562.9662.96
Random neurons, λ=0.05\lambda = 0.054.614.61M0.2050.2054.644.6459.3559.3563.0963.09
LoRA, r=16r = 16, all 2828 layers4.704.70M0.0180.0181.501.5058.5258.5262.5962.59
LoRA, r=32r = 32, 1414 layers4.704.70M0.0200.0201.511.5158.4358.4362.5862.58
LoRA, r=64r = 64, 77 layers4.704.70M0.0220.0221.541.5457.8857.8862.3762.37
LoRA, r=128r = 128, 44 layers5.375.37M0.0250.0251.611.6158.5458.5462.7062.70
  • Localization. Unlike LoRA, DAID aims to identify and localize parameters attributed with biased expression. The random-neuron control aims to compare this specifically, and shows the efficacy of optimizing targeted parameters. This holds at scale as well (see experiments done on Llama-3.1-8B\texttt{Llama-3.1-8B} presented in our response to Reviewer U1zz).
  • LoRA trains well with our objective and reaches a lower in-distribution MMPR (1.501.50–1.611.61 vs. 1.881.88), which confirms that the multigroup objective is effective. However, it does not generalize as well to other social benchmarks (StereoSet (race), BBQ (race/ethnicity) as in the paper and SoFa (race)). DAID achieves a higher ICAT and a better SS score on StereoSet, a more close to uniform distribution on out-of-distribution SoFa examples as well as comparable performance on BBQ.
Qwen2.5-1.5B, nationalityICAT ↑\uparrowSS ⋄\diamondsAMBs_\text{AMB} ⋄\diamondsDISs_\text{DIS} ⋄\diamondSoFa score ↓\downarrowSoFa MGD, NG=5N_G = 5 ↓\downarrowSoFa MGD, all 223223 ↓\downarrow
Base68.2468.2463.3163.312.502.502.732.730.2790.2790.5460.5461.4481.448
DAID80.53\mathbf{80.53}56.55\mathbf{56.55}0.29\mathbf{0.29}2.502.500.257\mathbf{0.257}0.360\mathbf{0.360}0.807\mathbf{0.807}
Random neurons, λ=0.2\lambda = 0.2 / 0.10.1 / 0.050.0573.973.9–74.974.959.859.8–60.360.31.451.45–2.592.594.134.13–4.364.360.2660.266–0.2680.2680.4590.459–0.4620.4621.1291.129–1.1371.137
LoRA, r=16r = 1677.1577.1558.2158.210.29\mathbf{0.29}2.792.790.2720.2720.3720.3720.8320.832
LoRA, r=32r = 3276.9676.9658.3258.321.051.053.433.430.2680.2680.3680.3680.8400.840
LoRA, r=64r = 6477.4377.4358.1158.110.810.812.38\mathbf{2.38}0.2720.2720.3960.3960.9330.933
LoRA, r=128r = 12875.3575.3559.2559.251.771.774.134.130.2700.2700.3770.3770.8420.842
  • Full fine-tuning was not feasible within the rebuttal period. LoRA at a matched budget is the closest comparison we could run. The LoRA results signal the possibility that full fine-tuning might not generalize as well as targeted training does. This remains an interesting experiment to conduct, and we will add this comparison in the final draft.
  • Changing λ\lambda does not affect random control outputs a lot. Its MMPR stays at 4.64.6–4.94.9 and its MMLU at 59.259.2–59.559.5. Its limit is therefore not the retention trade-off: randomly chosen neurons cannot express the debiasing update at any λ\lambda we tried, while the selected neurons reach 1.881.88 for 1.01.0 MMLU point.

3. "How were KK, λ\lambda and the number of epochs chosen … Was there a separate validation split?"

The hyperparameters were selected from a small grid search using the reported metrics, based on the validation split where the distractor groups were selected at random. The Table 4 presents the grid search for Llama-3.2-3B\texttt{Llama-3.2-3B}. Similarly, we do a manual grid search and the chosen parameters are presented in the Appendix in Table 6. The selected hyperparameters were selected to balance the tradeoff between model degeneracy and social bias performance gains.

References

[1] Marchiori Manerba, M., Stańczak, K., Guidotti, R., & Augenstein, I. (2024). Social bias probing: Fairness benchmarking for language models. EMNLP 2024.

[2] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. ICLR 2022.

Reviewer_ZwNwOpenReview ↗

Response to Authors

Thank you for the additional experiments. The held-out context/target results and the SoFa evaluation address my main concern about generalization, so I am raising my score.

Two points remain:

Contribution of neuron discovery. The parameter-matched LoRA baseline with the same objective performs comparably or better in distribution (MMPR 1.50 vs. 1.88). The OOD advantage of DAID is modest and shown for a single model and seed. This suggests the gains come largely from the multigroup objective, so I recommend framing neuron discovery as a secondary contribution. Hyperparameter selection. Please state clearly in the paper that the hyperparameters were selected on the evaluation split.

Please also include the held-out, SoFa, and LoRA results in the revision.

Reviewer_sSP5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DAID, a neuron-level debiasing framework for mitigating social biases across multiple demographic groups in large language models. The authors introduce Multigroup-StereoSet, which extends StereoSet by evaluating fixed completions across diverse demographic groups. DAID identifies MLP neurons whose suppression reduces the divergence from a uniform group distribution using a first-order approximation, and subsequently fine-tunes the corresponding parameters with a retention loss. Experiments on four language models demonstrate substantial reductions in group-probability disparities and improvements on conventional bias benchmarks, with moderate degradation in general model capabilities.

Critical strengths

1. Meaningful motivation and multigroup evaluation.\textbf{1. Meaningful motivation and multigroup evaluation.} The paper addresses an important limitation of conventional pairwise bias evaluation, which may overlook disparities across multiple demographic groups. Extending StereoSet with diverse comparison groups provides a useful setting for evaluating such disparities.

2. Promising debiasing performance and capability analysis.\textbf{2. Promising debiasing performance and capability analysis.} DAID achieves substantial reductions in multigroup probability disparities while updating only a small fraction of model parameters. The neuron ablation experiments and analyses of retention weights provide useful insights into the trade-off between bias mitigation and general model capabilities.

Critical weaknesses

1. Limited novelty of the multigroup formulation.\textbf{1. Limited novelty of the multigroup formulation.} Although the paper motivates multigroup bias evaluation, the proposed formulation appears closely related to existing multigroup fairness and counterfactual invariance objectives. In particular, demographic groups can be represented as categorical one-hot features, and the objective essentially encourages invariant conditional likelihoods across group identities. The paper does not sufficiently clarify how this formulation differs fundamentally from existing multigroup or distribution-alignment approaches, such as SoFa and Auto-Debias. Consequently, the novelty of the problem formulation remains somewhat unclear.

2. Insufficient isolation of the methodological contribution.\textbf{2. Insufficient isolation of the methodological contribution.} The proposed neuron attribution relies on a first-order activation-gradient approximation, while selective neuron fine-tuning is also closely related to existing methods. Although the paper includes a random-neuron baseline, it does not sufficiently distinguish the contribution of the proposed neuron selection strategy from that of the multigroup training objective. In particular, comparisons with full fine-tuning, parameter-matched LoRA, or alternative gradient-based neuron selection methods using the same multigroup objective are missing. This makes it difficult to determine whether the improvements specifically arise from DAID or primarily from the proposed training objective.

3. Limited generalization and robustness evaluation.\textbf{3. Limited generalization and robustness evaluation.} The experiments cover only nationality and occupation biases, English text, and models with at most 3B parameters. Furthermore, the number of demographic groups is fixed at five. While different group-selection strategies are used for training and evaluation, it is unclear whether the underlying contexts and completions are disjoint. Thus, the evaluation does not sufficiently establish generalization to unseen stereotypes, contexts, or larger numbers of demographic groups.

4. Incomplete fairness–utility trade-off analysis.\textbf{4. Incomplete fairness–utility trade-off analysis.} Although the paper analyzes the effects of neuron budgets and retention weights, the trade-off is mainly investigated within DAID. The degradation in general capabilities is non-negligible in some settings, particularly for occupation debiasing. Without comparing different methods under matched parameter budgets or comparable capability degradation, it remains unclear whether DAID achieves a better fairness–utility trade-off than alternative approaches.

Questions for authors

1. Multigroup formulation and novelty:\textbf{1. Multigroup formulation and novelty:} Could the authors clarify how the proposed multigroup objective fundamentally differs from existing categorical-group fairness or counterfactual invariance formulations? In particular, what distinguishes it from prior multigroup evaluation and distribution-alignment approaches beyond the construction of Multigroup-StereoSet?

2. Contribution of neuron selection:\textbf{2. Contribution of neuron selection:} Could the authors compare DAID against full fine-tuning, parameter-matched LoRA, and conventional gradient-based neuron selection methods using the same multigroup loss? Such comparisons would help isolate the benefit of the proposed neuron discovery procedure from the training objective itself.

3. Generalization and group scalability:\textbf{3. Generalization and group scalability:} Are the training and evaluation splits disjoint at the context and completion levels, or do they only differ in the selected demographic groups? Additionally, how does DAID perform when the number of groups increases beyond five, or when evaluated on previously unseen contexts and demographic groups?

4. Fairness–utility trade-off:\textbf{4. Fairness–utility trade-off:} Could the authors provide a Pareto analysis comparing multigroup bias reduction against MMLU/HellaSwag performance across different methods under comparable training budgets? This would help clarify whether DAID provides a more favorable trade-off rather than simply achieving stronger debiasing at the cost of general capabilities.

4 comments
AuthorsOpenReview ↗

Response By Authors (1/3)

We thank the reviewer for the valuable feedback, and for recognising the "meaningful motivation" of multigroup evaluation. Below, we address the weaknesses and questions raised by the reviewer.

1. "Could the authors clarify how the proposed multigroup objective fundamentally differs from existing categorical-group fairness or counterfactual invariance formulations?"

We operationalize on a multigroup framing of social bias, wherein a model that encoded social stereotypes would have a distribution over a set of demographies, which is farther away from uniform (which is when the model does not prefer a stereotypical generation conditioned on a group-indexed context). The Multigroup-StereoSet extension is curated to only have such examples where these group-indexed contexts allow for valid substitutions. In addition, we are also interested in localized edits on parameters of a model which are correlated with social-biased expression.

  • Auto-Debias [2] aligns the full masked-token distributions of cloze-style completions in purely encoder transformer models like BERT, while our debiasing technique is for decoder transformer models. Secondly, as mentioned, AutoDebias updates all the weights whereas we localize the edits needed in a model. Importantly, Auto-Debias takes a binary approach to identifying social bias, considering only two target groups, a stereotypical and an anti-stereotypical one.
  • SoFa [1] similarly to our paper discusses social bias as a multigroup objective. However, SoFa is a benchmark, while we propose a debiasing method. SoFa is a complementary dataset, but StereoSet allows for more natural sentences compared to the short assertions that SoFa contains. We provide the results on this benchmark as well in #3.
AuthorsOpenReview ↗

Response By Authors (2/3)

2. "Could the authors compare DAID against full fine-tuning, parameter-matched LoRA, and conventional gradient-based neuron selection methods using the same multigroup loss?"

We ran a parameter-matched LoRA [3] baseline and a random-neuron control at several λ\lambda, all with the same objective Lbias+λLret\mathcal L_{\text{bias}} + \lambda \mathcal L_{\text{ret}}, data and epochs:

Qwen2.5-1.5B, nationalitytrainable paramsMGD (SS) ↓\downarrowMMPR (SS) ↓\downarrowMMLU ↑\uparrowHSwag ↑\uparrow
Base–0.550.5523.0423.0459.2959.2963.5963.59
DAID, λ=0.2\lambda = 0.24.614.61M0.040.041.881.8858.2858.2861.9061.90
Random neurons, λ=0.2\lambda = 0.24.614.61M0.2080.2084.884.8859.2159.2163.1263.12
Random neurons, λ=0.1\lambda = 0.14.614.61M0.2080.2084.774.7759.4559.4562.9662.96
Random neurons, λ=0.05\lambda = 0.054.614.61M0.2050.2054.644.6459.3559.3563.0963.09
LoRA, r=16r = 16, all 2828 layers4.704.70M0.0180.0181.501.5058.5258.5262.5962.59
LoRA, r=32r = 32, 1414 layers4.704.70M0.0200.0201.511.5158.4358.4362.5862.58
LoRA, r=64r = 64, 77 layers4.704.70M0.0220.0221.541.5457.8857.8862.3762.37
LoRA, r=128r = 128, 44 layers5.375.37M0.0250.0251.611.6158.5458.5462.7062.70
  • Localization. Unlike LoRA, DAID aims to identify and localize parameters attributed with biased expression. The random-neuron control aims to compare this specifically, and shows the efficacy of optimizing targeted parameters. This holds for a larger size model as well (see experiments done on Llama-3.1-8B\texttt{Llama-3.1-8B} presented in our response to Reviewer U1zz).
  • LoRA trains well with our objective and reaches a lower in-distribution MMPR (1.501.50–1.611.61 vs. 1.881.88), which confirms that the multigroup objective is effective. However, it does not generalize as well to other social benchmarks (StereoSet (race), BBQ (race/ethnicity) as in the paper and SoFa (race)). It has a higher ICAT and a better SS score on StereoSet, a more close to uniform distribution on out-of-distribution SoFa examples as well as comparable performance on BBQ.
Qwen2.5-1.5B, nationalityICAT ↑\uparrowSS ⋄\diamondsAMBs_\text{AMB} ⋄\diamondsDISs_\text{DIS} ⋄\diamondSoFa score ↓\downarrowSoFa MGD, NG=5N_G = 5 ↓\downarrowSoFa MGD, all 223223 ↓\downarrow
Base68.2468.2463.3163.312.502.502.732.730.2790.2790.5460.5461.4481.448
DAID80.53\mathbf{80.53}56.55\mathbf{56.55}0.29\mathbf{0.29}2.502.500.257\mathbf{0.257}0.360\mathbf{0.360}0.807\mathbf{0.807}
Random neurons, λ=0.2\lambda = 0.2 / 0.10.1 / 0.050.0573.973.9–74.974.959.859.8–60.360.31.451.45–2.592.594.134.13–4.364.360.2660.266–0.2680.2680.4590.459–0.4620.4621.1291.129–1.1371.137
LoRA, r=16r = 1677.1577.1558.2158.210.29\mathbf{0.29}2.792.790.2720.2720.3720.3720.8320.832
LoRA, r=32r = 3276.9676.9658.3258.321.051.053.433.430.2680.2680.3680.3680.8400.840
LoRA, r=64r = 6477.4377.4358.1158.110.810.812.38\mathbf{2.38}0.2720.2720.3960.3960.9330.933
LoRA, r=128r = 12875.3575.3559.2559.251.771.774.134.130.2700.2700.3770.3770.8420.842
  • Full fine-tuning was not feasible within the rebuttal period. LoRA at a matched budget is the closest comparison we could run. The LoRA results signal the possibility that full fine-tuning might not generalize as well as targeted training does. This remains an interesting experiment to conduct, and we will add this comparison in the final draft.
AuthorsOpenReview ↗

Response By Comments (3/3)

3. "Are the training and evaluation splits disjoint at the context and completion levels … how does DAID perform when the number of groups increases beyond five, or when evaluated on previously unseen contexts and demographic groups?"

In the paper, the training and evaluation splits share contexts and completions and differ in the distractor groups (dispersed for training, random for evaluation). Following the reviewer's question, we built held-out splits and reran the search and training on their training portion only, with the paper's hyperparameters unchanged (K=3000K = 3000, λ=0.2\lambda = 0.2, 2020 epochs):

Multigroup-StereoSet Qwen2.5-1.5B\texttt{Qwen2.5-1.5B}MGD (SS), base →\rightarrow DAIDMMPR (SS), base →\rightarrow DAID
Held-out contexts: test0.510→0.1130.510 \rightarrow 0.11320.84→2.3920.84 \rightarrow 2.39
Held-out contexts: train0.527→0.0500.527 \rightarrow 0.05021.05→2.0021.05 \rightarrow 2.00
Held-out targets: test0.491→0.1220.491 \rightarrow 0.12223.76→2.7523.76 \rightarrow 2.75
Held-out targets: train0.533→0.0500.533 \rightarrow 0.05020.01→2.0020.01 \rightarrow 2.00

The held-out target groups do not appear in training, even as distractors. MMPR falls by 88%88\% on both unseen contexts and unseen groups, close to the 90%90\% on the training examples.

For more groups and unseen stereotypes, we evaluate on SoFa [1] (4,5524{,}552 stereotypes over 223223 identities). DAID is trained with NG=5N_G = 5, and we evaluate it with up to all 223223 identities per stereotype:

SoFa, Qwen2.5-1.5B\texttt{Qwen2.5-1.5B}SoFa score ↓\downarrowMGD, NG=5N_G = 5 ↓\downarrowMGD, NG=10N_G = 10 ↓\downarrowMGD, all 223223 ↓\downarrow
Base0.2790.2790.5460.5460.7220.7221.4481.448
DAID0.257\mathbf{0.257}0.360\mathbf{0.360}0.456\mathbf{0.456}0.807\mathbf{0.807}

The decrease in our metric holds as the number of groups increases. MGD decreases by 34%34\% at NG=5N_G = 5, 37%37\% at 1010, and 44%44\% over all 223223 identities.

4. "Could the authors provide a Pareto analysis comparing multigroup bias reduction against MMLU/HellaSwag performance across different methods under comparable training budgets?"

We do a small grid search across K/λK/\lambda hyperparameters, wherein we try to balance the model capability against social bias performance. The first table in #2 partly presents a performance analysis across different methods, which is presented here again succinctly.

MethodMMPR (SS) ↓\downarrowICAT ↑\uparrowMMLU ↑\uparrow
BiasEdit17.3817.3873.3673.3659.17\mathbf{59.17}
BBA48.7448.7477.3177.3155.7355.73
FBA121.13121.1368.5568.5546.3846.38
Random neurons (any λ\lambda above)4.64.6–4.94.973.973.9–74.974.959.2−59.5\mathbf{59.2-59.5}
DAID1.881.8880.53\mathbf{80.53}58.2858.28
LoRA with our objective1.50–1.61\mathbf{1.50–1.61}75.475.4–77.477.457.957.9–58.558.5

DAID is better than FBA and BBA on both axes. BiasEdit and the random control keep up to 11 more MMLU point, but are further from uniform distribution on the multigroup objective. The two methods trained with our objective, DAID and LoRA, present lower bias at a cost of about 11 MMLU point.

References

[1] Marchiori Manerba, M., Stańczak, K., Guidotti, R., & Augenstein, I. (2024). Social bias probing: Fairness benchmarking for language models. EMNLP 2024.

[2] Guo, Y., Yang, Y., & Abbasi, A. (2022). Auto-Debias: Debiasing masked language models with automated biased prompts. ACL 2022.

[3] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. ICLR 2022.

We thank the reviewer again for their careful reading and constructive feedback. Their comments have helped improved the paper. In response, we have added the Llama-3.1-8B\texttt{Llama-3.1-8B} experiments, the held-out and SoFa generalization studies, and the LoRA comparison. We are also currently working on clarifying the notation in the revised manuscript.

We hope our responses and the new results have addressed at least some of the reviewer’s concerns. If any points remain unclear, we would be glad to discuss them further during the discussion period.

Reviewer_U1zzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper provides various contributions:

  • Multi-group StereoSet, An extension of StereoSet that focuses on distractor groups
  • DAID, A two-stage debiasing method that performs selective fine-tuning on MLP neurons based on their effect of group distributions
  • An empirical finding on pairwise benchmarks

They evaluated the proposed DAID on 4 small open-source models across 2 demographic axes, and provided ablation studies on neuron selection and some of the parameters used in their approach, further validating their empirical findings and the strength of their approach. Overall, their empirical findings and the proposed method and dataset could reach some audience in the ICLR community. However, some clarifications on the theory, on some specific design choices and on the experiments are needed.

Critical strengths

The provided dataset addresses an existing problem from a different perspective. Although it extends an existing benchmark rather than introducing one from scratch, its contribution should be assessed based on the motivation and significance of the proposed improvements, as well as the additional insights enabled by the new dataset.

The proposed debiasing method also appears promising, although some clarification of specific design choices would help establish its soundness and overall value.

Critical weaknesses

My main concerns about the paper are about the scale and coverage of the study. Most models used here were relatively small models that can be mostly deemed outdated by curent standards. While I do not fundamentally disagree with the conclusions, it would have been more interesting to assess whether they hold with more recent models (especially on the zeroing and neuron suppressing part that can potentially be more "drastic" for bigger models), and under different conditions: more setups and more than 2 evaluation axes. Without it, it's hard to gauge whether it generalises and hard to gauge the relevance of these insights.

The other concerns are about soundness and methodological details. Some design choices and methodological details also remain unclear; therefore, the clarification requests that I have provided are important for my final assessment, as there might be some information that I have misunderstood during my evaluation of the paper.

Questions for authors

Clarification requests

  • Sec. 3 (first paragraph): I understand the metric as a measure of how unevenly a fixed continuation c is associated with different groups. However, is it therefore more precise to interpret it as a group-association disparity metric than a general measure of social bias? How does the metric distinguish a model that assigns low stereotypical probability uniformly across groups from one that assigns high sterotypical probability uniformly across groups, since both can produce low KL divergence?
  • Sec. 3.3: Have the authors quantified the approximation error or rank correlation between the first-order estimates and exact intervention-based marginal gains?
  • Based on Sec. 5 and Sec. 6.1 results: in most applications, those models are not used for text completion, but rather for their general-purpose capabilities. How would you quantify an acceptable degradation w.r.t curated benchmarks such as MMLU and HellaSwag?
  • Only the random neurons experiments and ablation studies show variation values. Are the results shown in some Tables completely seed-invariants (e.g, Table 2)? Which (hyper)parameters exactly influence variations on the results?
  • The models used in the study can be seen as outdated by current standards. Have the authors tried to assess if the empirical conclusions hold with more sota models?
  • App. (C.2): Can the authors provide some examples of "context associations which didn't allow for valid substitutions"?

Suggestions

  • (Ethics statement) : It should also be reported that a particular bias emerging from the simplification of social identities within the original dataset is probably involuntarily inherited from IngleHart and U.S Census occupation roles, as different classifications or local scientists would not necessarily entirely agree with the resulting classes
  • It would be interesting to provide as a future direction a hint on how to modify StereoSet or even Multi-GroupStereoSet to enable evaluation on other axes
  • The first-order approximation is central to the computational feasibility of the method, but suppressing a neuron changes its activation from aω to zero, which may not be a small perturbation. Could the authors validate their method against the exact marginal gains on a representative subset? In addition to absolute approximation error, rank correlation and top-k overlap would be particularly relevant because the approximation is used for neuron selection.
7 comments
AuthorsOpenReview ↗

Response By Authors (1/3)

We thank the reviewer for a careful reading of the paper, and for recognizing that the dataset "addresses an existing problem from a different perspective" and that the debiasing method "appears promising". We address issues and questions raised below.

1. Is it more precise to interpret the method as a group-association disparity metric? How does the method distinguish the possibilities of uniformly low v/s high probability assignment to completions since the KL divergence loss decreases in both the cases?

Indeed, our method and metric capture group-association disparity, which is a deliberate design choice. A model that assigns a stereotypical continuation the same probability for every group induces the uniform distribution over the groups. We consider that such a model does not exhibit social bias, whether that probability is high or low. Because the completion cc is held fixed while only the demographic group in the context xgx_g varies, the conditional probabilities P(c∣xg)P(c \mid x_g) are directly comparable across groups. We normalize the probability of the continuation over the groups (Eq. 1), so the distribution PgenP_{\text{gen}} over the group set GG, reflects only how this probability differs across groups, and not how large it is.

To address the reviewer's feedback, we verify whether DAID does not make stereotypical continuations more likely. We conducted an experiment which measured log⁡P(c∣xg)\log P(c \mid x_g) on Multigroup-StereoSet for the stereotypical (SS) and unrelated (UU, reference) continuations, over all 644×5644 \times 5 (example, group) pairs (Qwen2.5-1.5B\texttt{Qwen2.5-1.5B}, K=3000K = 3000). We record a relative drop in stereotypical associations in comparison to unrelated continuations: the mean Δlog⁡P(S)P(U)=−0.07\Delta \log \frac{P(S)}{P(U)} = -0.07. Stereotypical continuations therefore become slightly less likely relative to unrelated generations. Furthermore, we found that the target group's probability of a stereotypical continuation P(S)P(S) over the five selected groups (uniform: 0.20.2) falls from a mean of 0.3670.367 on the base model, with interquartile range [0.133,0.559][0.133, 0.559], to 0.199±0.0010.199 \pm 0.001, [0.170,0.223][0.170, 0.223]. The share of examples where the target exceeds uniform decreases from 62.4%62.4\% to 45.5%45.5\%. Since StereoSet matches each stereotypical continuation with its target group, this shows that DAID reduces the association with the group that the stereotype is about.

2. "Have the authors quantified the approximation error or rank correlation between the first-order estimates and exact … marginal gains?"

We compared our greedy search, which uses the first-order approx. (referred here as attr\texttt{attr}) with the same search using the exact marginal gain on GPT-2\texttt{GPT-2} (36,86436{,}864 neurons), where exact scoring is feasible. Other settings: 644644 examples from the nationality split, K=1200K = 1200, 5050 neurons per step.

  • DAID score: The score of the selected set is almost identical along the whole search (figure below).
  • Set after first iteration: the searches choose the same 4545 of 5050 neurons.
  • Final sets: they share 805805 of 1,2001{,}200 neurons (20.6×20.6\times random chance), and the selected neurons have the same distribution across layers.

Exact scoring needs one pass over the data per candidate per step. This is about 3737k candidates for GPT-2\texttt{GPT-2}, but 229229k–459459k for the models currently in the paper. Because a Monte Carlo estimate of the marginal gain is computationally expensive, we use a first-order Taylor expansion instead, which requires a single forward and backward pass per example, and in our experiments this remains close to the marginal gain in Eq. 3. However, we will add the analysis of the errors between these to search procedures in the final version of the manuscript.

the plots can be accessed using the anonymized links below.

daid_curves

layer_histogram

AuthorsOpenReview ↗

Response By Authors (2/3)

3. "How would you quantify an acceptable degradation w.r.t. … MMLU and HellaSwag?"

We believe that there is no single objective threshold for an acceptable degradation. The trade-off between debiasing and general model capabilities is well documented in prior work [10]. We compare the effects of our method to the state-of-the-art debiasing methods, reporting both bias and model performance metrics.

In DAID, the parameter λ\lambda controls this trade-off. The grid search over a small set of K/λK/\lambda hyper-parameter choices allows us to select conditions wherein the loss in general capability can be balanced against gains observed in social bias performance. We will report the full K/λK/\lambda dependence for every model in the final revision of the paper. We have on Llama-3.2-3B\texttt{Llama-3.2-3B} (K=6000K = 6000; base MMPR 23.223.2, MMLU 55.3755.37) as reported in the paper the dependence on the λ\lambda which weighs the retention loss Lbias+λLret\mathcal L_{\text{bias}} + \lambda \mathcal L_{\text{ret}}:

λ\lambda110.10.10.010.01
MMPR ↓\downarrow3.223.221.511.511.261.26
MMLU ↑\uparrow (change)54.8954.89 (−0.5-0.5)53.8653.86 (−1.5-1.5)45.9845.98 (−9.4-9.4)

4. "Are the results … seed-invariant? Which (hyper)parameters influence variations?"

  • Neuron search: deterministic given the data, KK and step size. We perform neuron search without sampling due to the data scarcity.
  • Targeted training: the example order is the only source of randomness in our DAID results and is thus seed-dependent. We have worked on adding this experiment below, and we will incorporate it in the final version of the paper.

Below, we present the results of the DAID training with three seeds (Qwen2.5-1.5B\texttt{Qwen2.5-1.5B}, nationality):

MGD (SS) ↓\downarrowMGD (AA) ↓\downarrowMMPR (SS) ↓\downarrowMMPR (AA) ↓\downarrow
Base0.550.550.420.4223.0423.0412.2312.23
DAID (33 seeds)0.044±0.0010.044 \pm 0.0010.045±0.0010.045 \pm 0.0011.92±0.021.92 \pm 0.021.93±0.031.93 \pm 0.03

5. "Most models used here were relatively small … especially on the zeroing and neuron suppressing part"

We agree with the reviewer that testing our method on a bigger model is a valuable experiment. We have conducted recent experiments on Llama-3.1-8B\texttt{Llama-3.1-8B}. We will add these results to the final version of the paper. Below we present results for one setting (nationality axis, λ=0.1\lambda = 0.1):

Llama-3.1-8BICAT ↑\uparrowsAMBs_\text{AMB} ⋄\diamondsDISs_\text{DIS} ⋄\diamondBOLD Comp. / Regard ↑\uparrowMGD (SS) ↓\downarrowMMPR (SS) ↓\downarrowHSwag ↑\uparrowMMLU ↑\uparrow
Base66.0066.001.741.74−1.63-1.630.380.38 / 0.520.520.550.5520.5220.5276.5276.5263.7463.74
Prefix prompting55.7555.753.203.20−0.23-0.230.380.38 / 0.410.410.520.5217.9717.97––
SelfDebias74.4274.42−3.66-3.66−4.51-4.51–0.100.102.482.48––
BiasEdit64.4764.47−0.17-0.17−2.17-2.170.370.37 / 0.510.510.460.4613.1713.1775.3275.3263.5063.50
FBA79.3779.371.981.98−0.70-0.700.170.17 / 0.260.260.580.5829.0629.0673.2573.2544.3544.35
BBA71.4571.450.580.580.17\mathbf{0.17}0.140.14 / 0.200.200.490.4916.3916.3973.5373.5358.2958.29
DAID, K=6000K = 600078.2978.290.09\mathbf{0.09}−3.60-3.600.410.41 / 0.550.550.006\mathbf{0.006}1.29\mathbf{1.29}72.9672.9661.1961.19
DAID, K=9000K = 900079.3879.380.200.20−3.60-3.600.42\mathbf{0.42} / 0.56\mathbf{0.56}0.004\mathbf{0.004}1.25\mathbf{1.25}73.4073.4060.2460.24
DAID, K=12000K = 1200082.25\mathbf{82.25}0.840.84−3.40-3.400.42\mathbf{0.42} / 0.56\mathbf{0.56}0.005\mathbf{0.005}1.25\mathbf{1.25}72.6472.6460.1460.14

We observe that the paper's conclusions hold for this model as well:

  • DAID decreases multigroup social bias while also performing superior on the standard StereoSet benchmark. The performance on BOLD similarly does not decrease.
  • The cost in general capability loss remains similar or lower compared to other debiasing methods.

6. "… examples of 'context associations which didn't allow for valid substitutions'?"

We performed a manual review of the dataset and removed the examples in which the continuation is no longer plausible once the target group is substituted, or in which the target term does not refer to a social group. This removed 131131 nationality and 7777 occupation examples. Representative cases (substitution in square brackets):

  1. "The Pope lives in Rome, which is in Italy" [… in Singapore] (the true fact is tied to the target.)
  2. "The Colombian is a paper in Washington." (the term is not a social group.)
  3. "The prosecutor spoke up and asked for maximum punishment" [The roofer …]. (the role in the sentence is tied to the occupation)
AuthorsOpenReview ↗

Response By Authors (3/3)

7. "… more setups and more than 2 evaluation axes …" (addressing a weakness pointed)

We agree that more bias axes would strengthen the paper. However, we have deliberately decided to leave this extension for future work. The limiting factor for this is data. As discussed under #6, the constructions need natural contexts that stay coherent when the group is substituted, and completions that do not encode the group:

  • StereoSet [1]. In 79.3%79.3\% (192/242192/242) of gender examples, the completion contains a gendered word ("She is …"), so substitution would test grammatical agreement. The religion split has 7878 examples over 33 targets, one of which ("Bible") is not a group. Even on the usable axes, filtering keeps 66%66\% of nationality and 31%31\% of occupation examples.
  • Other resources. CrowS-Pairs [2] is a set of single-sentence minimal pairs, in which the group and the stereotype appear in the same sentence. There is no completion separate from the group mention, so substituting a group rewrites the very sentence being scored, and its likelihood then mixes the bias with the fluency of the edit.
    • This is also why Multigroup-StereoSet builds on StereoSet's intersentence format (rather than the intrasentence split). The group appears only in the context, and the completion carrying the stereotype stays fixed across groups.
    • BOLD [3] has no stereotype labels, and SoFa [4] uses strict templated assertions. BOLD scores free-form generations, so there is no fixed stereotypical completion whose likelihood can be compared across groups. SoFa’s short identity-stereotype templates have limited context and no anti-stereotypical counterpart. We therefore use SoFa for external evaluation only, not for constructing the multigroup objective. We would appreciate any pointers to relevant datasets, including data for further bias axes that we could use in this work.

Following Reviewer's ZwNw comment, we added a new experiment within the scope of this discussion, which shows the extent to which our work extends to unseen distractor groups. We split the data into the train and test splits where the test split has unseen contexts and unseen distractor groups.

  • Unseen contexts: MMPR falls from 20.8420.84 to 2.392.39.
  • Unseen target groups (absent from training even as distractors): MMPR falls from 23.7623.76 to 2.752.75.

8. Suggestions

  • Inherited classifications. We agree, and will add to the ethics statement that the distractor groups inherit the Inglehart–Welzel map's country-level cultural zones [5] and the U.S. Census occupation hierarchy [6]. Other classifications would yield different distractor sets.
  • New axes. DAID is defined for any NG≥2N_G \ge 2. A new axis needs substitutable contexts with group-neutral completions. For gender, this requires generating new data, which we intend to do synthetically with automatic and manual filtering. Kindly refer to point 7 for additional context and reasons behind data scarcity.
  • Exact marginal gains. See point 2.

References

[1] Nadeem, M., Bethke, A., & Reddy, S. (2021). StereoSet: Measuring stereotypical bias in pretrained language models. ACL-IJCNLP 2021.

[2] Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020). CrowS-Pairs: A challenge dataset for measuring social biases in masked language models. EMNLP 2020.

[3] Parrish, A., Chen, A., Nangia, N., et al. (2022). BBQ: A hand-built bias benchmark for question answering. Findings of ACL 2022.

[4] Dhamala, J., Sun, T., Kumar, V., et al. (2021). BOLD: Dataset and metrics for measuring biases in open-ended language generation. FAccT 2021.

[5] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2018). Gender bias in coreference resolution: Evaluation and debiasing methods. NAACL 2018.

[6] Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018). Gender bias in coreference resolution. NAACL 2018.

[7] Marchiori Manerba, M., Stańczak, K., Guidotti, R., & Augenstein, I. (2024). Social bias probing: Fairness benchmarking for language models. EMNLP 2024.

[8] World Values Survey Association & European Values Study (2023). The Inglehart–Welzel World Cultural Map.

[9] U.S. Census Bureau (2020). Occupation Classification Codes: 2018 Census Occupation Classification.

[10] Meade, N., Poole-Dayan, E. & Reddy, S. (2022). An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models ACL 2022

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.