- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DISCO, a framework for identifying skill-specific attention heads in vision-language models. The method estimates head contributions through counterfactual activation patching across different answer formats and image styles, then selects heads based on their relative contributions across four visual reasoning skills. The authors introduce separability and composability metrics and evaluate the selected heads on both single-skill and two-skill tasks across three VLMs.
Critical strengths
The paper addresses a worthwhile question: identifying components that matter for a task is insufficient to establish that they are specific to the underlying skill. Explicitly considering effects on other skills is a useful direction. I also appreciate the distinction between preceding and final token positions, the question-level identification/evaluation split, and the detailed intervention results in the appendix. The selection ablations provide encouraging evidence that cross-skill information helps identify more selective head sets.
Critical weaknesses
-
My main concern is the composability metric does not establish the property that the paper claims to measure. Equations (9)–(11) compare the joint intervention effect against averages of individual or mixed-random intervention effects. A positive score does not require both skill-specific head sets to contribute. For example, suppose ablating (H_A) changes 50% of originally correct predictions, ablating (H_B) has no effect, and ablating their union still changes 50%. If the random controls have no effect, both the absolute and relative composability scores are 25 percentage points, even though (H_B) contributes nothing. Conversely, if two components are both necessary and removing either one already causes failure, joint ablation may add no further effect. I would therefore be cautious about interpreting these measurements as evidence of compositional computation. Controlled interventions that independently change the two task factors, or restoration experiments linking each head set to its expected subcomputation, would provide more convincing evidence.
-
The proposed scoring procedure averages effects across several surface forms, but averaging does not ensure that an individual head contributes consistently across them. A head with a strong effect in one format can still receive a high pooled score. Filtering for originally correct predictions may also change the relative representation of each surface form.
-
Subtracting a random-head baseline is a reasonable attempt to account for differences in task sensitivity. However, the resulting score measures selectivity relative to that baseline. In this case, it does not directly establish that the selected heads have small absolute effects on non-target tasks.
Questions for authors
See Weaknesses
3 comments
Q3. Subtracting a random-head baseline when computing the separability score does not directly establish that the selected heads have small absolute effects on non-target tasks.
A3. Absolute effects alone may be poor metrics, since tasks may have different sensitivity to intervention on random heads. As an example, disrupting a set of heads may cause the performance to drop 50% on Task A and 70% on Task B. This may suggest the heads affect Task B more. However, due to different sensitivity, disrupting an equal number of random heads may degrade performance by 10% and 50% on the two tasks, respectively. Subtracting these random baselines yields adjusted effects of 40% and 20%, and reveals a stronger contribution to the Task A / Skill A. The larger absolute effects of these heads on Task B do not necessarily indicate greater contribution to Skill B, but rather higher sensitivity of Task B to interventions.
Thank you for your valuable and responsible review, which has been very inspiring for us.
We would appreciate it if you could let us know whether our response has addressed your concerns. We look forward to further discussion with you.
We thank the reviewer for recognizing that our paper addresses a worthwhile question and for appreciating our separable head selection method, the distinction between different token positions, and the selection ablations. Below, we address the reviewer’s concerns. All the experimental results reported below have been incorporated into the latest version of the paper.
Q1. The composability metric does not establish the property that the paper claims to measure...if two components are both necessary and removing either one already causes failure, joint ablation may add no further effect.
A1. We thank the reviewer for pointing out the limitations of our original composability metrics. We revise Equations (9)–(10) to define composability as the joint intervention effect minus the maximum of individual or mixed-random intervention effects. Under this definition, a positive score requires heads of both skills to make contributions. In the reviewer's example, under the new metric, the composability score becomes 0 instead of 25. As shown in the tables below, our method still achieves the highest average composability across all three models (7.82, 9.68, and 3.90), substantially outperforming the strongest baselines (1.81, 2.48, and 2.71). These results demonstrate that our method identifies more composable attention heads.
In practice, disrupting the heads for one skill alone usually has limited effects rather than causing the corresponding multi-skill task to fail completely. For example, heads that contribute to Skill A but primarily support other skills are not selected for Skill A. However, these heads may still serve as weak substitutes, partially supporting Skill A when heads that we identified are disrupted.
Moreover, even if disrupting the heads for one skill causes complete failure on the corresponding multi-skill task, there are two possible cases. First, these heads may support necessary basic functions shared across skills (e.g., output formatting), as existing methods only focus on heads with the largest effects on the target skill. Such heads do not reflect the skill composability we aim to identify. Therefore, assigning them a low score under our metric is reasonable. Second, these heads may genuinely support the distinctive function of the target skill, which is the case the reviewer is concerned about. As shown in Table 2, the baselines achieve very low separability, which suggests the first case is true because basic functional heads have substantial effects on multiple skills.
On Qwen2.5-VL:
On Qwen3-VL:
On Gemma4:
Q2. The proposed scoring procedure averages effects across several surface forms, but averaging does not ensure that an individual head contributes consistently across them.
A2. We agree with the reviewer that simply averaging may mask variance. That is, a head may be important for some surface forms but not others and still gets identified.
That is exactly why we introduced the coverage mechanism in head identification (Lines 228–238). Coverage is defined as the proportion of instances for which the head receives a positive score, i.e., the head contributes to that instance. The final score is the product of the average score and coverage. In this way, a head can only be identified if it has a high average score and contributes to many data points, which employ different surface forms.