- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Causal Darwinian Head Selection (CDHS), a diagnostic framework that ranks attention heads by how much task loss goes up when they are ablated. It looks at cumulative head removal, class-specific specialization, and how head importance emerges during training. Experiments on Tiny-ImageNet and CIFAR-10/100 suggests that a small subset of heads contribute disproportionately to performance, and that broad semantic specialization recurs across training seeds. However, missing experimental details and not enough differentiation from existing attribution methods weakens the contribution.
Critical strengths
Section 4.2.2 keeps the examples used for ranking heads seperate from the held-out evaluation, which reduces the risk that the ablation results are just reflecting the ranking data.
Figure 14 compares individual rankings against conditional greedy selection and beam search. The loss-based results show non-additive interactions and acknowledge the limitations of interpreting heads independantly.
Figure 15 looks at five training seeds and finds recurring animal, vehicle, and household preferences. This supports broad semantic specialization beyond what is observed in a single trained model.
Critical weaknesses
The paper does not adequately discuss recent related work, and it compares only against random ablation, with no recent head-attribution baselines. The methodological contribution is not clearly distinguished from CIGMA [1], which is not cited. Both methods ablate individual attention heads, average the resulting changes over examples, rank the heads, and evaluate selected subsets. CIGMA measures reductions in background information gain, while CDHS measures increases in task loss, so the objectives are different. Still, the underlying intervention-and-ranking procedures are similar, and CDHS doesnt clearly show what methodological advance it offers beyond changing the attribution target and adding specialization analyses.
The experiments are not specified well enough. Appendix A.2.1 describes the computing infrastructure but leaves out essential model and training details, like the architecture configuration, learning rate, batch size, and training duration. The sizes of the ranking and held-out splits are also unclear. These omissions makes reproduction impossible and make the reported behavior hard to put in context.
The specialization claims in the main text go beyond the cross-seed evidence. Section 4.2.4 puts emphasis on fine-grained specialists, but Figure 15 reports fine-grained evidence of only 0.01–0.05 and explicitly admits limited support for a recurring, strongly single-class-selective cluster. The conclusions should separate the supported recurrence of broad semantic preferences from the weaker evidence for fine-grained specialization.
Reference:
[1] Maisha Maliha and Dean F. Hougen. CIGMA: Causal Information-Gain Mechanistic Attribution of Attention Heads in Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 9891–9900.
Questions for authors
How does CDHS go beyond CIGMA’s causal head-ablation and ranking procedure, and what evidence shows the value of the proposed distinction?
Can you provide the full architecture, training configuration, ranking/evaluation split sizes, and seed counts for each experiment?
How do you reconcile the fine-grained specialist claims with the limited cross-seed support in Figure 15?