- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CogVSR, which decomposes spatial reasoning into eight annotated functions, and uses activation probes to identify associated attention heads across three VLM families. The authors report sparse head–function associations and examine their behavioral relevance through head suppression, activation steering, and spatial input augmentation.
Critical strengths
- CogVSR could be a useful resource beyond aggregate accuracy evaluation. Verification of the decompositions and function labels by multiple annotators also strengthens the dataset construction.
- The analysis combines observations across models with behavioral interventions.
- In addition to attribution maps, it compares targeted head suppression with suppression of an equal number of random heads and reports positive activation interventions. These experiments provide evidence that the selected heads are behaviorally relevant.
Critical weaknesses
- The probe may capture task-related shortcuts rather than distinct cognitive functions. Answer format, vocabulary, and subquestion position may predict the labels. Appending the shared layer-average activation also makes attribution to individual heads ambiguous.
- Behavioral relevance does not establish functional specialization. Cross-function interventions and layer-matched random controls are needed to demonstrate selectivity. Likewise, fewer heads exceeding an attribution threshold do not establish a spatial-capacity bottleneck.
- The SHA protocol leaves a potential information-leakage concern unresolved. Its prompt includes {answer}, but the source is unspecified. Ground-truth answers could introduce unavailable test-time information; even without them, external localization assistance offers an alternative explanation for the gains.
- Evaluation independence needs clarification. The “out-of-domain” benchmarks also supply CogVSR’s source data. Deduplication and the splits used for head selection, intervention-direction estimation, and hyperparameter tuning are insufficiently documented. Furthermore, evaluations use only 200 samples per dataset without confidence intervals or paired tests. The small downstream gains lack uncertainty estimates.
Questions for authors
1, Which splits determine the heads, intervention directions, and α? How are downstream questions, images, and scenes deduplicated against CogVSR? 2. What supplies {answer} in SHA, and do the gains persist when augmentation uses only the image and question?
5 comments
Q3. Information leakage in the SHA protocol.
Source of {answer}, and the evaluated model never sees it. The {answer} field is the CogVSR reference subanswer. Boxes and masks are produced by a separate model (Gemini 2.5-Flash), which uses this field only to list the object names to localize (Appendix I), while the evaluated VLM receives only the edited image and the question. The overlay carries only object names and positions, not the answer: the subquestions ask for relations, directions or truth values, which the model must still compute.
To further address the concern, we regenerate all boxes and masks from only the image and the question, with the answer removed from the SHA prompt. +BBox+Mask still yields consistent gains over the original input: spatial accuracy improves by +10.30 on Intern2B and +5.13 on Llama90B, and relational accuracy by +7.58 and +4.77, respectively. Compared with the answer-guided setting, performance drops only slightly, and the question-only setting retains 83%--91% of the original gains across all metrics. +BBox+Mask also remains the best variant in all 8 settings. These results confirm that the improvement mainly comes from better spatial grounding rather than answer-guided target selection.
Table. Effect of spatial input augmentation with boxes and masks generated from the image and question only (no answer in the SHA prompt).
| Model | Input | Spatial llm | Spatial acc | Relational llm | Relational acc |
|---|---|---|---|---|---|
| Intern2B | Original | 61.37 | 56.82 | 63.25 | 58.13 |
| +BBox | 65.12 | 58.31 | 64.20 | 61.48 | |
| +Mask | 59.24 | 59.10 | 66.03 | 55.92 | |
| +Color | 64.31 | 59.62 | 65.04 | 60.57 | |
| +Shape | 67.42 | 63.85 | 67.91 | 64.63 | |
| +Color+Shape | 68.33 | 65.18 | 68.12 | 65.04 | |
| +BBox+Mask | 70.46 | 67.12 | 68.94 | 65.71 | |
| Llama90B | Original | 86.21 | 82.07 | 88.03 | 83.92 |
| +BBox | 89.63 | 86.34 | 89.41 | 85.22 | |
| +Mask | 87.18 | 83.02 | 90.73 | 84.61 | |
| +Color | 88.57 | 84.41 | 89.30 | 85.18 | |
| +Shape | 90.94 | 86.58 | 91.22 | 87.83 | |
| +Color+Shape | 91.47 | 86.79 | 91.56 | 88.12 | |
| +BBox+Mask | 92.31 | 87.20 | 91.88 | 88.69 |
Q4. Evaluation independence, data splits, and uncertainty.
Independence. The 200 evaluation questions per benchmark are disjoint from the 400 questions per benchmark used to build CogVSR (deduplicated by question ID). 0–24% of the evaluation images also appear in CogVSR, but results on the image-disjoint subset change by at most 0.61 points. We agree that "out-of-domain" is the wrong term and will use "held-out benchmark questions." This setting tests transfer across task format: heads and directions are derived from decomposed subquestions with gold context, and applied while the model answers the original question end to end.
Splits. None of the benchmark evaluation questions is used for head selection, direction estimation, or the choice of α (Table 2). The probe and steering directions use the CogVSR training split. Head importance, elbow selection, and α = 0.1 use the CogVSR test split; α is chosen once on Qwen2.5-VL-7B and then fixed for all models and benchmarks. Selecting heads with importance computed on the training split instead keeps 78–86% of the heads and changes the Table 2 results by at most 1.68 points (Table 3).
Table 2. Data used at each step of the analysis.
| Step | Data | Source in paper |
|---|---|---|
| Probe training | CogVSR train (909 / 2,969), correct answers only | Sec. 5.2 |
| Head importance & elbow selection | CogVSR test (233 / 790) | Sec. 6.1, Fig. 2, App. K |
| Ablation (Tables 2, 7) | CogVSR test | Sec. 6.2 |
| Steering directions | CogVSR train | Sec. 6.5 |
| Choice of α = 0.1 | CogVSR test, spatial / relational | App. M, Table 11 |
| Benchmark evaluation | disjoint from CogVSR | Sec. 6.5 |
Table 3. Intervention results (%, llm / acc) of Qwen2.5-VL-7B on the CogVSR test set, with cognitive heads selected using importance computed on the test split (as in Table 2) or on the training split.
| Function | Random | Cognitive (test-split selection) | Cognitive (train-split selection) | Heads shared |
|---|---|---|---|---|
| Spatial | 80.04 / 75.92 | 57.69 / 53.42 | 58.83 / 54.61 | 83% |
| Relational | 82.11 / 77.86 | 63.85 / 59.57 | 64.92 / 60.71 | 81% |
| Low-Level | 84.39 / 80.21 | 47.92 / 43.18 | 49.51 / 44.86 | 78% |
| High-Level | 88.27 / 84.10 | 68.31 / 63.04 | 69.04 / 63.85 | 86% |
| Info | 90.02 / 85.73 | 73.77 / 68.55 | 74.62 / 69.30 | 80% |
| Recall | 88.39 / 84.03 | 71.23 / 65.98 | 72.15 / 66.87 | 84% |
| Math | 82.76 / 78.44 | 62.84 / 57.63 | 63.98 / 58.80 | 79% |
| Decision | 86.51 / 82.19 | 59.41 / 54.22 | 60.47 / 55.31 | 85% |
About Evaluation, we would like to clarify:
- Consistency. All 12 held-out cells improve (one-sided sign test, p = 2.4×10⁻⁴), as do 32 of all 36 cells. Cells sharing a model are not fully independent, so we treat this as a consistency check.
- Dose–response with sign reversal. In Table 11, every positive α (0.05–0.3) raises both spatial perception and relational reasoning, every negative α lowers both, and the drop grows with |α|. Noise would not produce such a sign-symmetric, ordered pattern. The effect peaks at α = 0.1 and remains positive up to 0.3.
- Necessity. Masking the same heads removes 58–73% of benchmark accuracy (Table 4).
Paired tests. Pooling the four benchmarks (n = 800 paired samples per model), the gains are positive for all three models but not individually significant:
| Model | Δ | McNemar p | 95% CI (paired bootstrap) |
|---|---|---|---|
| Qwen7B | +0.75 | 0.23 | [−0.46, 1.94] |
| Llama11B | +0.53 | 0.38 | [−0.67, 1.73] |
| Intern8B | +0.68 | 0.27 | [−0.52, 1.87] |
We would like to highlight that the intervention is a single training-free linear shift on a small set of heads, with one α = 0.1 shared across all models and benchmarks and never tuned on any evaluation benchmark. It is designed to test direction, not to maximize accuracy. Even so, it consistently improves spatial reasoning across models and benchmarks, showing that the identified heads are not only correlated with spatial reasoning but also serve as a practical lever for improving it. This opens directions for future work, such as head-targeted fine-tuning, input-adaptive steering strengths, and combining head steering with training-based methods to obtain larger gains. We will tone down the claim accordingly in the revised version.
We deeply appreciate the time and effort you have dedicated to reviewing our work. Below, we address the questions you raised.
Q. Does the probe capture surface shortcuts rather than function?
Surface form and modality. We analyze surface-form variation across the eight functions. Token lengths show wide, overlapping distributions, and the word clouds differ only in content words that reflect what each function asks, not in phrasing templates or structure (Fig. 2-3, rebuttal PDF). Regarding modality, as described in [Section 4.1], some functions (e.g., Low-level and High-level Visual Perception) naturally rely on vision, while others relate more to language; this tendency is intrinsic to the cognitive processes rather than an artifact of our pipeline. Token position is also controlled in Table 7, Appendix). Together, these results indicate that the function groups are not driven by trivial lexical, structural, or modality artifacts. The causal evidence comes from the intervention experiments in Sections 6.2 and 6.5.
Why we add the layer-averaged vector. Prior work suggests that cognitive functions vary with layer depth [1], so two heads in different layers with similar activation patterns may still play distinct roles. We therefore include the layer-averaged vector as a coarse signal of each head's layer context. Table 7 (Appendix) compares selection with and without it: masking heads selected with layer information lowers spatial-perception accuracy to 57.69, versus 63.92 without it and 80.04 for random heads. The layer information thus sharpens the selection, while heads selected without it still clearly outperform random heads.
[1] Zheng, Z., et al. "Attention heads of large language models: A survey. arXiv 2024." arXiv preprint arXiv:2409.03752.
Q. Selectivity of the identified heads and the spatial-capacity claim.
Cross-function ablations demonstrate selectivity. We ablate the heads of each function on Qwen2.5-VL-7B and evaluate performance on all eight functions (Fig. 1). In every row (8/8), ablating a function's own heads causes the largest drop. For example, ablating the spatial heads lowers spatial performance to 57.7, whereas ablating any other function's heads leaves it between 69.5 and 75.4.
Layer-matched random controls. We further sample random heads with the same per-layer distribution as each cognitive head set (Table 1). Layer matching makes random heads only slightly more harmful (by 1.7–2.6 points), while the cognitive heads remain 16.6–24.6 points below the layer-matched control. Layer position therefore explains only a small part of the effect.
Table 1. Cross-function ablation on Qwen2.5-VL-7B (LLM-judge accuracy, %). Rows: masked head set; columns: evaluated function.
Spatial-capacity claim. We agree that fewer heads above a threshold does not by itself establish a bottleneck, and we do not make this causal claim: Section 6.3 states that the scarcity "may be associated with weaker performance on spatial reasoning." We will align two stronger sentences in Section 6 and the Introduction with this wording. The threshold is descriptive and matches the elbow of the importance curves (Fig. 14); all interventions use elbow-selected heads.