acceptodds
Withdrawn from ICLR 2027

Causal Darwinian Head Selection: A Mechanistic Framework for Interpreting Attention Heads

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding how internal components of Transformer models contribute to prediction remains a central challenge for mechanistic interpretability. We propose Causal Darwinian Head Selection (CDHS), a Darwinian, causal intervention framework for analyzing attention heads as a population of computational units. CDHS uses temporary ordered ablations to estimate head-level causal contribution and to measure the resulting changes in model behavior; it is intended as a post hoc diagnostic method. Applied to Transformer-based image classifiers, CDHS jointly characterizes three aspects of head organization: sparse causal fitness, class-conditional behavioral effects, and checkpoint-wise emergence under task-aligned supervision. We find that a small subset of dominant heads contributes disproportionately to predictive performance, whereas temporarily intervening on many low-fitness heads has limited immediate effect. Interventions on dominant heads induce structured class-level shifts in prediction behavior, suggesting that some heads support class-discriminative computations rather than only global accuracy. Using checkpoint-wise analysis and a matched label-permutation control, we further observe that fitness concentration strengthens during training under real labels more than under permuted labels. These results position CDHS as a framework for linking causal intervention, class-level model behavior, and training dynamics in vision Transformers.

Primary area:
interpretability and explainable AI
Keywords:
Attention Heads, Causal Darwinian Head Selection, Mechanistic Framework

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_RPAAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Causal Darwinian Head Selection (CDHS), a diagnostic framework that ranks attention heads by how much task loss goes up when they are ablated. It looks at cumulative head removal, class-specific specialization, and how head importance emerges during training. Experiments on Tiny-ImageNet and CIFAR-10/100 suggests that a small subset of heads contribute disproportionately to performance, and that broad semantic specialization recurs across training seeds. However, missing experimental details and not enough differentiation from existing attribution methods weakens the contribution.

Critical strengths

Section 4.2.2 keeps the examples used for ranking heads seperate from the held-out evaluation, which reduces the risk that the ablation results are just reflecting the ranking data.

Figure 14 compares individual rankings against conditional greedy selection and beam search. The loss-based results show non-additive interactions and acknowledge the limitations of interpreting heads independantly.

Figure 15 looks at five training seeds and finds recurring animal, vehicle, and household preferences. This supports broad semantic specialization beyond what is observed in a single trained model.

Critical weaknesses

The paper does not adequately discuss recent related work, and it compares only against random ablation, with no recent head-attribution baselines. The methodological contribution is not clearly distinguished from CIGMA [1], which is not cited. Both methods ablate individual attention heads, average the resulting changes over examples, rank the heads, and evaluate selected subsets. CIGMA measures reductions in background information gain, while CDHS measures increases in task loss, so the objectives are different. Still, the underlying intervention-and-ranking procedures are similar, and CDHS doesnt clearly show what methodological advance it offers beyond changing the attribution target and adding specialization analyses.

The experiments are not specified well enough. Appendix A.2.1 describes the computing infrastructure but leaves out essential model and training details, like the architecture configuration, learning rate, batch size, and training duration. The sizes of the ranking and held-out splits are also unclear. These omissions makes reproduction impossible and make the reported behavior hard to put in context.

The specialization claims in the main text go beyond the cross-seed evidence. Section 4.2.4 puts emphasis on fine-grained specialists, but Figure 15 reports fine-grained evidence of only 0.01–0.05 and explicitly admits limited support for a recurring, strongly single-class-selective cluster. The conclusions should separate the supported recurrence of broad semantic preferences from the weaker evidence for fine-grained specialization.

Reference:

[1] Maisha Maliha and Dean F. Hougen. CIGMA: Causal Information-Gain Mechanistic Attribution of Attention Heads in Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 9891–9900.

Questions for authors

How does CDHS go beyond CIGMA’s causal head-ablation and ranking procedure, and what evidence shows the value of the proposed distinction?

Can you provide the full architecture, training configuration, ranking/evaluation split sizes, and seed counts for each experiment?

How do you reconcile the fine-grained specialist claims with the limited cross-seed support in Figure 15?

Reviewer_p9i3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper seeks to understand and characterize the effect of small sets of specialized attention heads on vision transformer models doing image classification. The authors propose a method that assigns a score to each head based on how important it is for the classification task, and then show that highest-scoring heads indeed harm performance if ablated. Furthermore, the paper studies the progressive emergence of specialization during training.

Critical strengths
  • The paper is for the most part well-written and easy to access.
  • Studying head specialization dynamics during training is interesting and, to the best of my knowledge, a novel topic for vision transformers.
Critical weaknesses
  • Unclear framing. The Darwinian framing is confusing. The authors score heads individually and then show that high-scoring heads matter for the task, but it is unclear what role Darwinian theory plays, since nothing resembling natural selection takes place. Darwinian selection requires variation, inheritance, and differential reproduction, and none of these is present: heads do not reproduce, and the model is frozen during the analysis. The authors themselves state that "selection" only means ranking, which conflicts with the claim of evidence "beyond metaphor" in Section 2.3.
  • The experimental setting is underspecified. The authors mention training a vision transformer, but no other details on the architecture are given. At least, the paper should report the size of the model, whether the training happens from scratch or from an existing checkpoint, training hyperparameters and data. In addition, the model appears quite weak, as the reported accuracy on Tiny ImageNet (approximately 35%) is far from values previously reported in literature (e.g., see [a]).
  • Limited evaluation. The paper presents results limited in scope, both in terms of models studied and evaluation datasets. The only model analyzed is a vision transformer trained by the authors, and it is unclear how the results translate to standard off-the-shelf vision transformers such as those of the original ViT paper, or the backbone of CLIP, for example. Regarding data, evaluation is again limited, comprising results only on Tiny ImageNet and CIFAR.
  • Limited novelty and missing related literature. The central claim that attention heads in vision transformers acquire highly specialized, non-interchangeable roles is not novel in the field. Prior works (e.g. [b,c,d]) have already shown that heads in ViTs can exhibit specific, interpretable roles, and that ablating small, specific sets heads can drastically impact downstream performance. This stream of prior works should at least be acknowledged in the manuscript, and the authors should position their work against it.
  • Unconvincing results. Section 4.2.1 could probably be removed and its results absorbed into 4.2.2, as the experiment is identical, and the evidence is much stronger when head scoring and evaluation are run on separate splits of Tiny ImageNet. Moreover, the training dynamics results, which are the most novel experimental part of the paper, offer limited support for the claims. The differences between real-label training and the permutation control are small and overlap across seeds. In addition, the permuted-label model cannot generalize, so fitness measured on validation data may not be comparable between the two conditions.

References:

[a] Gani et al., How to Train Vision Transformer on Small-scale Datasets?, BMVC 2022

[b] Gandelsman et al., Interpreting CLIP's Image Representation via Text-Based Decomposition, ICLR 2024

[c] Basile et al., ResiDual Transformer Alignment with Spectral Decomposition, TMLR 2025

[d] Lu et al., Discovering Attention Head Interactions in Vision Transformers, CVPR Findings 2026

Questions for authors
  • Does the zero-ablation happen during the forward pass? And do the authors have any way of understanding whether it is measuring direct or indirect effects (i.e., mediated by downstream attention/MLP blocks)?
  • What does "temporary" mean in line 16? And what does "checkpoint-wise emergence under task-aligned supervision" mean in line 20/21?
  • Why is only the positive causal contribution of a head considered when computing its fitness? How can it be excluded that a head is causally harming the model on a certain dataset?
  • How is the class-conditional fitness score (line 262) computed? In the paper I can only find a global fitness score defined.
  • In section 4.2.3, what are the classes that exhibit stronger/weaker drops in accuracy? It would be informative to read their actual names rather than class IDs to understand if there is any interpretable pattern.
  • In Figure 4, how are different niches defined?
Reviewer_Tk5LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper applied the common ablation procedure on individual attention heads in transformer for the image classification task and provided interpretation for the ablation procedure from the Darwinian procedure, i.e., variation and selection. The experiments showed ablation results showed a small set of attention heads which have high causal impacts on the classification tasks. The authors also demonstrated some attention heads are specialized to certain classes or groups of the classes, and show the changes in the sparsity through the training.

Critical strengths
  1. The paper provided an interesting interpretation of ablation analyses used in mechanistic interpretability research from Darwinian theory.
  2. The paper not only showed the strong ablation effects over a sparse set of heads at the fully-trained models but also showed the changes in the sparsity during the training process.
  3. The paper clearly stated the ablation procedure clearly in the method section.
Critical weaknesses
  1. The paper's contribution is limited although the paper is mainly selling their methodology contribution.

Based on my reading, the procedure described in the method section is the standard and widely-adopted ablation procedure for circuit tracing in the mechanistic interpretability research [1][2]. And the main methodology contribution is the interpretation of the ablation from the Darwinian perspective, which I feel like is limited and may not actually bring practical insights to mech interp community. It would be better for the authors to explain the main contribution of the paper.

  1. The task which the paper studied is relevantly simple.

The paper studies the classic image classification tasks on ImageNet, which has been well studied. I would extend to more complex tasks in either vision domain (VLMs) or language domain (LLMs).

  1. The ablation only shows the necessity of the attention heads. It would be good if the authors could also try some activation patching methods like causal mediation analysis (CMA) [3] to find the heads which are sufficient for the task.

[1] Olsson, Catherine, et al. "In-context learning and induction heads." arXiv preprint arXiv:2209.11895 (2022). [2] Wu, Wenhao, et al. "Retrieval head mechanistically explains long-context factuality." International Conference on Learning Representations. Vol. 2025. 2025. [3] Wang, Kevin, et al. "Interpretability in the wild: a circuit for indirect object identification in GPT-2 small." arXiv preprint arXiv:2211.00593 (2022).

  1. Some minor issues in the paper presentation.

First, the figures lack necessary explanations in the captions and the text font in the figures is relatively small. For Figure 3, it is better to use simpler labels rather than "Class_01443537".

Second, a few figures are not that informative. It is hard to infer the points made by the authors from the figures. It would be better demonstrated by a in a clearer way. For example, I failed to reach the same conclusion in (Line 362-366) from Figure 3. It would be better if the authors could elaborate on this clearer.

Third, I would not introduce the methodology again in the experiment sessions (Line 251-256).

Questions for authors
  1. What is the size of the model trained for image classification task?
  2. How do the Tiny ImageNet results differ from the CIFAR results reported in the appendix? The results in appendix seem like a simple transfer on the new dataset without highlighting any new findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.