- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose the Causal Competition Framework (CCF), which quantifies a transformer attention head’s importance by the squared Wasserstein-2 distance between activation distributions of the intact and head-ablated model. They wrap this measurement in an evolutionary algorithm, Representation-level Darwinian Head Competition (RDHC), where heads in a single transformer layer are “individuals” that reproduce proportional to their fitness, with offspring inheriting parent weights plus Gaussian mutation. The empirical section reports three findings on DBpedia text classification (with AG News and Yahoo! Answers in the appendix): (1) head importance under Wasserstein ranking is heavy-tailed, (2) heads specialize by semantic class, and (3) a pairwise “rescue” analysis, in which one head is ablated and another is amplified, reveals pair-specific compensation structure that survives after controlling for marginal row and column effects. An appendix provides formal properties of the Wasserstein criterion, the population-update protocol, and additional experimental results.
Critical strengths
- The pairwise rescue analysis with row-column residual decomposition is a genuinely new contribution. Prior work on head importance has focused on either individual head ablation or coarse pruning. The directional pairwise rescue experiment, in which ablation of head d is partially compensated by amplifying head r by a tuned factor, is a novel direction.
- The theoretical appendix cleanly separates general discrepancy properties from conditional geometric results, with instructive counterexamples.
- The method is honest about what the math does and does not entail. (e.g. “a large removal effect need not make replication beneficial” (3.2), “this quantity alone does not establish functional cooperation or redundancy” (3.2), “Darwinian specialization additionally requires lineage-frequency changes” (3.4).)
- Calibration-evaluation split is cleanly handled.
Critical weaknesses
- The Darwinian framing does no empirical work. The inheritance result holds under neutral replication (\beta = 0), so selection pressure contributes nothing.
- Heavy-tailed head importance and class specialization are well-established findings. The paper does not benchmark Wasserstein fitness against existing importance measures (gradient attribution, accuracy-drop, activation patching). Does it find the same important heads as less expensive techniques or are they different?
- Experimental setup is under-specified: no model architecture, layer, N, T, \sigma, \beta, run count, or evaluation set size in the main text. Some of these details are banished to the appendix, but they are important for reproducibility.
- The joint-ablation interaction is defined and analyzed theoretically but never measured.
- Six generations is too few to support “evolutionary timescales” language.
- Three near-identical English topic-classification benchmarks is a weak generalization claim.
Questions for authors
- Does \beta > 0 produce measurably different outcomes than \beta = 0?
- How does Wasserstein ranking correlate with standard importance measures?
- Why is J_{i,j} defined but never reported?