acceptodds
Under review as a conference paper at ICLR 2027

When Speed Costs Accuracy: Causal Faithfulness of Component Attribution for Gender-Bias Localization

Abstract

There are mechanistic techniques that aim to compute the effect of individual components or groups of components on a certain type of output. Moreover, these methods are widely used to target the components that produce stereotypical behavior. Sometimes we want affordable solutions instead of generic tasks, such as IOI, Greater-Than, for quick results – and for this, we need to understand whether cheap surrogates (e.g., DLA, EAP-IG) actually recover causal importance for bias specifically, and how much of this behavior is caused by the indirect effect that is undetectable through these techniques. In our work, we benchmark four attribution methods: DLA, AtP, EAP-IG, DE against activation patching ground truth for gender-bias localization across three models: GPT-2 XL, Llama-3.2-1B, Gemma-2-2B under two ablation baselines on a gender-focused StereoSet variant. We found that the frozen-norm approximation is nearly free – it accounts for only under a sixth of the error, and removing it leaves every ranking metric unchanged. The indirect effects account for the whole problem – 88-110% of the total – leaving direct attribution comparable to or worse than a constant-zero predictor, and recovering only about half of patching's ten most important components. A single backward pass recovers the rest: attribution patching finds 7-9 of that top ten at three forward-equivalent passes rather than 234-1248. The difference does not survive the intervention: ablating each method's selected components at matched capability cost removes a comparable share of the measured bias regardless of which method chose them, so ranking components by individual effect does not determine how good they are as a set. Faithful single-component attribution is therefore necessary but not sufficient for localization: choosing a set of components needs a set-level objective, which none of these methods provide. We report stereotype score and language-modeling scores alongside a continuous bias measure, since a mean logit difference can be driven to zero by shrinking every logit.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Societal Bias, Gender Bias, LLM, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hSonOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is considering the problem of gender-bias localisation (whatever the authors mean by that as it is not defined in abstract nor in the intro) and compare different (all existing and no novelty regarding this point) interpretability methods on the StereoSets dataset. Five such methods are compared in the paper. Overall I find this paper very poorly written with rather complex and vague sentences. The authors do mention that they used Claude to make the paper "clear" but I am afraid this seems to have made it pretty unclear and difficult for even an expert such as myself who has been working in this topic for over a decade to understand the value of this work

Critical strengths

A comparison of five (all existing) localisation methods on a single dataset.

Critical weaknesses
  • The paper does not provide a clear definition of "localisation of gender-bias". Do you mean across countries, cultures or languages? Or just finding which part of a neural net bias neurones are located? I guessed the latter from the context but that point should have been made clear at the beginning of the paper.
  • The writing is rather sloppy/informal at times and then there are pretty complex sentences with many clauses that are difficult to even understand to even a native English expert.

For example, look at the following sentences.

  1. "Recent advancements in mechanistic interpretability (MI) have provided methods that allow people to identify the components underlying desired behavior and then edit them." -→ "people"?
  2. "To understand the differences in the use of these methods for the gender bias localization task, we benchmarked them against an ablation ground truth on a gender StereoSet1 variant." → "ablation ground truth" means what? "these methods", "them" refers to what?
  3. "The results are that direct attribution does not recover causal importance for gender bias: its error is 88–110% of ∑︁ |AP|, no better than a constant-zero predictor, and it recovers four or five of patching’s ten most important components with the sign right little better than chance." ... that is a pretty dense and complex sentence. Even after reading for three times I still could not understand what you want to say.
  • The experimental results are reported on a single dataset (StereoSets) which is a templated dataset. It is not clear whether the results would generalise to other datasets.
Questions for authors
  • Sterosets is a templated dataset and does not reflect true social bias variations. Why did you use this dataset? What other datasets can you report results on? We cannot make reliable conclusions on a single dataset.
  • The paper considers only binary gender. why?
  • how did you decide on those five methods to compare?
Reviewer_dPKtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the causal faithfulness of efficient attribution methods for identifying components associated with gender bias in language models. They compare 4 attribution methods with activation patching across three language models on a dataset. The experiments show that most attribution errors are due to indirect effects. More accurate component rankings do not lead to better bias mitigation.

Critical strengths
  1. Understanding the interpretability and model debiasing of language models is an important topic.
  2. The paper compares multiple attribution methods across three model architectures and several evaluation metrics. The experiment design is relatively good.
  3. The mathematical derivation and empirical validation are sound and provide useful insights into why direct attribution methods differ from activation patching.
Critical weaknesses
  1. The inline reference format is not aligned with ICLR’s requirements. The reference should use \citep.
  2. The language models used in this study are outdated and very small. It is not guaranteed whether the experiment findings can transfer to other larger LLMs. Experiments on most recent LLMs would be necessary, given that some of the biases may not exist anymore in the latest LLMs. Also, evaluating on one dataset is also not enough for a comprehensive analysis and to conclude the paper. It is unclear to me whether the observed errors and mitigation behavior can extend to other forms of social bias or LLMs.
  3. Experimental results are inconsistent between the main page and appendix. Section 6.1 reports 75, 77, and 86 evaluation units for the three models, but Appendix Table 2 and Section A.14 report 527, 533, and 622. How many evaluation units does the paper evaluate? Besides, Section 6.3 reports frozen-norm errors ranging from 4.4% to 14.5%, but Table 3 reports 3.9% to 13.6%. Section 6.2 also reports different ranking correlations from Table 2. The discrepancies need further explanation.
  4. Section 7.1 reports the maximum bias reduction across different values of k within each capability budget. I wonder how this best k is selected? Using a validation set or other analysis?
  5. LLM rephrasing needs further validation. The StereoSet is rewritten by an LLM. However, the paper misses validation of whether the rephrased examples preserve the original meaning and whether the retained examples are representative of the original benchmark. Additional human evaluation is needed.
Questions for authors
  1. It would be better to report the actual runtime and GPU memory usage of the proposed method.
  2. Appendix A.1: The effect of Gemma-2's final-logit soft-cap is reported separately, but I could not find the results anywhere. Where is this analysis presented?
  3. Line 168 Typo: Duplicated wording: Eq. equation 3.
Reviewer_89AvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies cheaper alternatives to activation patching for circuit analysis and causal attribution in a gender bias setting. They look at direct logit attribution (DLA), direct effect (DE), attribution patching (AtP), and a particular edge attribution patching w/ integrated gradients (EAP-IG) methods. They evaluate on StereoSet over GPT-2 XL, Gemma-2-eB, and Llama-3.2-1B models. The authors find that attribution patching best matches activation patching among the cheaper methods. DLA and DE perform worse as they miss the effects that an early part of the model can have on later layers. Recalculating final normalization alone does little to improve their rankings. The authors additionally find that more accurate individual scores do not consistently lead to better bias reduction at a similar cost to language performance in this setting. They conclude that scoring individual components accurately does not provide a reliable signal for their efficacy at the group component level.

Critical strengths

The comparisons done by the paper is extensive across methods and covers multiple models. Separating direct and indirect effects does seem to reveal a clear distinction in attribution method capabilities, at least in their alignment to activation patching. The main result is that most of DLA and DE's disagreement with AP comes from indirect effects, which the two methods do not calculate. The reported pattern is consistent across all three models.

Critical weaknesses
  1. I found the text nearly impossible to parse and reads closer to an internal report of results. Many acronyms and study numbers were introduced before their meanings (e.g., DLA and EAP-IG in the abstract, DLA, DE, and AP in the introduction, and some range of 234-1248 components in the abstract). Also the abstract speaks of 110% of the problem being explained by indirect effects, without proper explanation of how a >100% effect would be meaningful in the studies. In section 3.2, "we call this a knockout" doesn't seem to have a clear referent for "this". In Section 4.3, the categorization of frozen norm as "Source A" and indirect effect as "Source B" adds needless terminology. In Section 6.1, Metrics, the paper says a debiasing edit "turns on" the question of component direction accuracy, but does not properly explain of elaborate on this. The results section could sorely use more tables and/or figures to summarize results. There are paragraphs of nearly impossibly densely reported numbers, with insufficient framing for clear interpretation. All of this makes the paper of little value to the academic community, as the results of any valid experiments are not communicated with the level clarity necessary for the time available to the typical academic reader.

    1a. Marginally related, equations are labeled "Eq. equation #" throughout the manuscript.

  2. The paper does not engage fully with the existing literature. Its framing of the methods inaccurately suggest that the explored alternatives were all (or nearly all) developed as a cheaper alternative to activation patching. To my understanding of the literature, this historical account is not accurate of Direct Effect and Direct Logit Attribution methods, and only indirectly accurate to EAP-IG. In addition, the paper cites Blodgett et al. (2020) regarding categories of harm, but fails to follow the norm set by that paper of specifying the type of harm that is being addressed in one's research questions with appropriate accounting in the experimental setup. At the very least, an argument for why the authors disagree to the necessity of this norm in this paper is necessary. Finally, there has been major criticisms to the validity of data in StereoSet ([1] Blodgett et al. 2021) which should be addressed when the dataset is used at this point. Either an argument for why the dataset is still appropriate for investigating the paper's research questions, or alternatively, using (or at least adding) a dataset that is more well-regarded in the literature for evaluation.

[1] Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R., & Wallach, H. (2021). Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 1004–1015). Association for Computational Linguistics.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.