acceptodds
Under review as a conference paper at ICLR 2027

Attention in Space: Functional Roles of VLM Heads for Spatial Reasoning

Abstract

Despite remarkable advances in large Vision-Language Models (VLMs), spatial reasoning remains a persistent challenge. In this work, we investigate how VLMs represent and perform spatial reasoning through a mechanistic interpretability lens. We introduce CogVSR, a structured diagnostic framework that decomposes complex spatial reasoning tasks into compositional, step-by-step subquestions designed to simulate human-like reasoning, each aligned with specific cognitive functions such as spatial perception. This enables fine-grained analysis of how spatial reasoning is organized within model internals. Building on CogVSR, we develop a probing framework to identify and characterize attention heads specialized for these functions. Our analysis across diverse VLM families reveals that these functional heads are universally sparse, vary in number and distribution across functions. Interestingly, spatially specialized heads are fewer than those for other cognitive functions, highlighting their scarcity. We propose methods to better disentangle spatial-related functions to improve spatial understanding, which is consistent with the activation of latent spatial heads. Intervention experiments further demonstrate their critical role in spatial reasoning: removing functional heads leads to performance degradation, while emphasizing them enhances accuracy. This study provides new interpretability driven insights into how VLMs attend to space and paves the way for enhancing complex spatial reasoning in multimodal models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial reasoning, functional heads

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wesbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CogVSR, which decomposes spatial reasoning into eight annotated functions, and uses activation probes to identify associated attention heads across three VLM families. The authors report sparse head–function associations and examine their behavioral relevance through head suppression, activation steering, and spatial input augmentation.

Critical strengths
  1. CogVSR could be a useful resource beyond aggregate accuracy evaluation. Verification of the decompositions and function labels by multiple annotators also strengthens the dataset construction.
  2. The analysis combines observations across models with behavioral interventions.
  3. In addition to attribution maps, it compares targeted head suppression with suppression of an equal number of random heads and reports positive activation interventions. These experiments provide evidence that the selected heads are behaviorally relevant.
Critical weaknesses
  1. The probe may capture task-related shortcuts rather than distinct cognitive functions. Answer format, vocabulary, and subquestion position may predict the labels. Appending the shared layer-average activation also makes attribution to individual heads ambiguous.
  2. Behavioral relevance does not establish functional specialization. Cross-function interventions and layer-matched random controls are needed to demonstrate selectivity. Likewise, fewer heads exceeding an attribution threshold do not establish a spatial-capacity bottleneck.
  3. The SHA protocol leaves a potential information-leakage concern unresolved. Its prompt includes {answer}, but the source is unspecified. Ground-truth answers could introduce unavailable test-time information; even without them, external localization assistance offers an alternative explanation for the gains.
  4. Evaluation independence needs clarification. The “out-of-domain” benchmarks also supply CogVSR’s source data. Deduplication and the splits used for head selection, intervention-direction estimation, and hyperparameter tuning are insufficiently documented. Furthermore, evaluations use only 200 samples per dataset without confidence intervals or paired tests. The small downstream gains lack uncertainty estimates.
Questions for authors

1, Which splits determine the heads, intervention directions, and α? How are downstream questions, images, and scenes deduplicated against CogVSR? 2. What supplies {answer} in SHA, and do the gains persist when augmentation uses only the image and question?

5 comments

We deeply appreciate the time and effort you have dedicated to reviewing our work. Below, we address the questions you raised.

Q. Does the probe capture surface shortcuts rather than function?

Surface form and modality. We analyze surface-form variation across the eight functions. Token lengths show wide, overlapping distributions, and the word clouds differ only in content words that reflect what each function asks, not in phrasing templates or structure (Fig. 2-3, rebuttal PDF). Regarding modality, as described in [Section 4.1], some functions (e.g., Low-level and High-level Visual Perception) naturally rely on vision, while others relate more to language; this tendency is intrinsic to the cognitive processes rather than an artifact of our pipeline. Token position is also controlled in Table 7, Appendix). Together, these results indicate that the function groups are not driven by trivial lexical, structural, or modality artifacts. The causal evidence comes from the intervention experiments in Sections 6.2 and 6.5.

Why we add the layer-averaged vector. Prior work suggests that cognitive functions vary with layer depth [1], so two heads in different layers with similar activation patterns may still play distinct roles. We therefore include the layer-averaged vector as a coarse signal of each head's layer context. Table 7 (Appendix) compares selection with and without it: masking heads selected with layer information lowers spatial-perception accuracy to 57.69, versus 63.92 without it and 80.04 for random heads. The layer information thus sharpens the selection, while heads selected without it still clearly outperform random heads.

[1] Zheng, Z., et al. "Attention heads of large language models: A survey. arXiv 2024." arXiv preprint arXiv:2409.03752.

Q. Selectivity of the identified heads and the spatial-capacity claim.

Cross-function ablations demonstrate selectivity. We ablate the heads of each function on Qwen2.5-VL-7B and evaluate performance on all eight functions (Fig. 1). In every row (8/8), ablating a function's own heads causes the largest drop. For example, ablating the spatial heads lowers spatial performance to 57.7, whereas ablating any other function's heads leaves it between 69.5 and 75.4.

Layer-matched random controls. We further sample random heads with the same per-layer distribution as each cognitive head set (Table 1). Layer matching makes random heads only slightly more harmful (by 1.7–2.6 points), while the cognitive heads remain 16.6–24.6 points below the layer-matched control. Layer position therefore explains only a small part of the effect.

Table 1. Cross-function ablation on Qwen2.5-VL-7B (LLM-judge accuracy, %). Rows: masked head set; columns: evaluated function.

Masked headsSpatialRelationalHigh-levelDecision
Spatial57.6977.3981.2680.34
Relational73.9863.8581.5576.54
High-level69.4974.0468.3177.86
Decision72.4473.6182.0859.41
Random (same n)80.0482.1188.2786.51
Random, layer-matched77.4680.4085.7883.96

Spatial-capacity claim. We agree that fewer heads above a threshold does not by itself establish a bottleneck, and we do not make this causal claim: Section 6.3 states that the scarcity "may be associated with weaker performance on spatial reasoning." We will align two stronger sentences in Section 6 and the Introduction with this wording. The threshold is descriptive and matches the elbow of the importance curves (Fig. 14); all interventions use elbow-selected heads.

Q3. Information leakage in the SHA protocol.

Source of {answer}, and the evaluated model never sees it. The {answer} field is the CogVSR reference subanswer. Boxes and masks are produced by a separate model (Gemini 2.5-Flash), which uses this field only to list the object names to localize (Appendix I), while the evaluated VLM receives only the edited image and the question. The overlay carries only object names and positions, not the answer: the subquestions ask for relations, directions or truth values, which the model must still compute.

To further address the concern, we regenerate all boxes and masks from only the image and the question, with the answer removed from the SHA prompt. +BBox+Mask still yields consistent gains over the original input: spatial accuracy improves by +10.30 on Intern2B and +5.13 on Llama90B, and relational accuracy by +7.58 and +4.77, respectively. Compared with the answer-guided setting, performance drops only slightly, and the question-only setting retains 83%--91% of the original gains across all metrics. +BBox+Mask also remains the best variant in all 8 settings. These results confirm that the improvement mainly comes from better spatial grounding rather than answer-guided target selection.

Table. Effect of spatial input augmentation with boxes and masks generated from the image and question only (no answer in the SHA prompt).

ModelInputSpatial llmSpatial accRelational llmRelational acc
Intern2BOriginal61.3756.8263.2558.13
+BBox65.1258.3164.2061.48
+Mask59.2459.1066.0355.92
+Color64.3159.6265.0460.57
+Shape67.4263.8567.9164.63
+Color+Shape68.3365.1868.1265.04
+BBox+Mask70.4667.1268.9465.71
Llama90BOriginal86.2182.0788.0383.92
+BBox89.6386.3489.4185.22
+Mask87.1883.0290.7384.61
+Color88.5784.4189.3085.18
+Shape90.9486.5891.2287.83
+Color+Shape91.4786.7991.5688.12
+BBox+Mask92.3187.2091.8888.69

Q4. Evaluation independence, data splits, and uncertainty.

Independence. The 200 evaluation questions per benchmark are disjoint from the 400 questions per benchmark used to build CogVSR (deduplicated by question ID). 0–24% of the evaluation images also appear in CogVSR, but results on the image-disjoint subset change by at most 0.61 points. We agree that "out-of-domain" is the wrong term and will use "held-out benchmark questions." This setting tests transfer across task format: heads and directions are derived from decomposed subquestions with gold context, and applied while the model answers the original question end to end.

Splits. None of the benchmark evaluation questions is used for head selection, direction estimation, or the choice of α (Table 2). The probe and steering directions use the CogVSR training split. Head importance, elbow selection, and α = 0.1 use the CogVSR test split; α is chosen once on Qwen2.5-VL-7B and then fixed for all models and benchmarks. Selecting heads with importance computed on the training split instead keeps 78–86% of the heads and changes the Table 2 results by at most 1.68 points (Table 3).

Table 2. Data used at each step of the analysis.

StepDataSource in paper
Probe trainingCogVSR train (909 / 2,969), correct answers onlySec. 5.2
Head importance & elbow selectionCogVSR test (233 / 790)Sec. 6.1, Fig. 2, App. K
Ablation (Tables 2, 7)CogVSR testSec. 6.2
Steering directionsCogVSR trainSec. 6.5
Choice of α = 0.1CogVSR test, spatial / relationalApp. M, Table 11
Benchmark evaluationdisjoint from CogVSRSec. 6.5

Table 3. Intervention results (%, llm / acc) of Qwen2.5-VL-7B on the CogVSR test set, with cognitive heads selected using importance computed on the test split (as in Table 2) or on the training split.

FunctionRandomCognitive (test-split selection)Cognitive (train-split selection)Heads shared
Spatial80.04 / 75.9257.69 / 53.4258.83 / 54.6183%
Relational82.11 / 77.8663.85 / 59.5764.92 / 60.7181%
Low-Level84.39 / 80.2147.92 / 43.1849.51 / 44.8678%
High-Level88.27 / 84.1068.31 / 63.0469.04 / 63.8586%
Info90.02 / 85.7373.77 / 68.5574.62 / 69.3080%
Recall88.39 / 84.0371.23 / 65.9872.15 / 66.8784%
Math82.76 / 78.4462.84 / 57.6363.98 / 58.8079%
Decision86.51 / 82.1959.41 / 54.2260.47 / 55.3185%

About Evaluation, we would like to clarify:

  • Consistency. All 12 held-out cells improve (one-sided sign test, p = 2.4×10⁻⁴), as do 32 of all 36 cells. Cells sharing a model are not fully independent, so we treat this as a consistency check.
  • Dose–response with sign reversal. In Table 11, every positive α (0.05–0.3) raises both spatial perception and relational reasoning, every negative α lowers both, and the drop grows with |α|. Noise would not produce such a sign-symmetric, ordered pattern. The effect peaks at α = 0.1 and remains positive up to 0.3.
  • Necessity. Masking the same heads removes 58–73% of benchmark accuracy (Table 4).

Paired tests. Pooling the four benchmarks (n = 800 paired samples per model), the gains are positive for all three models but not individually significant:

ModelΔMcNemar p95% CI (paired bootstrap)
Qwen7B+0.750.23[−0.46, 1.94]
Llama11B+0.530.38[−0.67, 1.73]
Intern8B+0.680.27[−0.52, 1.87]

We would like to highlight that the intervention is a single training-free linear shift on a small set of heads, with one α = 0.1 shared across all models and benchmarks and never tuned on any evaluation benchmark. It is designed to test direction, not to maximize accuracy. Even so, it consistently improves spatial reasoning across models and benchmarks, showing that the identified heads are not only correlated with spatial reasoning but also serve as a practical lever for improving it. This opens directions for future work, such as head-targeted fine-tuning, input-adaptive steering strengths, and combining head steering with training-based methods to obtain larger gains. We will tone down the claim accordingly in the revised version.

Reviewer_tEP8OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CogVSR, a diagnostic dataset decomposing 1,142 spatial-reasoning questions into 3,759 subquestions annotated with eight cognitive functions (Sec. 4). A probing framework identifies attention heads associated with each function across six VLMs from three families (Sec. 5), finding functional heads are sparse and spatial heads especially scarce (Sec. 6.1–6.3). Head ablations, an input-induction method (SHA), and activation-shift interventions argue for their causal role (Sec. 6.2–6.5).

Critical strengths

CogVSR is a useful artifact: cognition-inspired decomposition with two-stage human verification (28.7% filtered, Sec. 4.3) yields interpretable subQAF triples beyond prior token-level attention analyses.

Consistent sparsity findings across InternVL, Qwen-VL, and Llama-Vision at 2B–90B scales (Fig. 2, Appendix F) make the "sparse functional heads" claim credible.

Critical weaknesses

Selection bias in probing (Sec. 5.2). Probes are trained only on correctly answered subquestions, so "cognitive heads" are partly "heads active when the model succeeds". An analysis including incorrect responses is needed to isolate function from correctness.

SHA confound (Sec. 6.4, Table 3). SHA injects bounding boxes/masks from Gemini2.5-Flash; the ~10% gain on InternVL3-2B may reflect externally provided localization rather than "activation of latent spatial heads". Oracle-box and random-box baselines would disentangle mechanism from information leak.

Weak positive-intervention effect (Sec. 6.5, Table 5). Gains are mostly <2% in-domain and <1% OOD, on only 200 samples, with no significance tests — weak support for "emphasizing heads enhances accuracy" (abstract).

Correlational core claim (Sec. 6.3). "Spatial heads are scarce, hence VLMs fail at spatial reasoning" is never causally tested (e.g., does fine-tuning increase their count, and does that improve reasoning?).

Questions for authors

What is the inter-annotator agreement (e.g., Cohen's κ) in the two-stage CogVSR verification, and how were multi-label conflicts resolved?

If probes are trained including incorrectly answered subquestions, do the same heads emerge? How much of Table 2's effect survives?

For SHA, what accuracy do ground-truth boxes (vs. Gemini-predicted, vs. random boxes) yield? Does the head-activation increase (Fig. 4) persist with oracle boxes?

Are the Table 5 improvements statistically significant across seeds/sample draws? How sensitive are they to α beyond Appendix M?

Why is the 0.001 importance threshold meaningful, and how stable are head counts to that choice (Sec. 6.1)?

2 comments

Thank you very much for your time and effort in reviewing our paper. We respond to each point below; related weaknesses and questions are answered together.

Q. Selection bias in probing.

When the probe is trained on all subquestions, including incorrect ones, largely the same heads emerge, and 88–92% of Table 2's effect survives.

Why we filter. When a subquestion is answered incorrectly, we cannot tell whether the labelled function was carried out at all. Including such samples adds label noise to the localization step. The filter is there to keep the labels valid.

Probe trained on all subquestions. We retrain the probe on all subquestions and repeat the masking experiment of Table 2 on Qwen2.5-VL-7B. From the following Table, we observe that:

  • The filter yields more precise heads. Masking the heads selected with the filter ("Correct only") causes larger drops on all eight functions than masking heads selected from all subquestions.
  • Same heads emerge: 76–85% of the original cognitive heads are selected again, and the head-importance maps are highly correlated (r = 0.85–0.91).

Table. Effect of the probe training data on cognitive-head intervention results for Qwen2.5-VL-7B. "Correct only" trains the probe on correctly answered subquestions; "All" trains it on all subquestions. Each cell reports llm / acc after masking heads. Lower values indicate stronger intervention effects. The last two columns give the share of the original cognitive heads that are selected again and the Pearson correlation between the two head-importance maps.

FunctionRandomCognitive (Correct only)Cognitive (All)Heads unchangedImportance-map r
Spatial80.04 / 75.9257.69 / 53.4260.12 / 55.8476%0.88
Relational82.11 / 77.8663.85 / 59.5765.47 / 61.2379%0.86
Low-Level84.39 / 80.2147.92 / 43.1850.86 / 46.3185%0.91
High-Level88.27 / 84.1068.31 / 63.0470.05 / 64.9283%0.90
Info90.02 / 85.7373.77 / 68.5575.63 / 70.4880%0.87
Recall88.39 / 84.0371.23 / 65.9872.81 / 67.5578%0.85
Math82.76 / 78.4462.84 / 57.6364.37 / 59.2082%0.89
Decision86.51 / 82.1959.41 / 54.2261.93 / 56.7484%0.90

Q. About SHA gain.

SHA is an input-induced method: the accuracy gain (Table 3) and the change in head importance (Fig. 4) are both induced by the input and co-occur. The causal evidence is in Section 6.5, where the heads are modified directly with the input unchanged. We further clarify that the gain is not explained by leaked information (InternVL3-2B, LLM judge):

  • No answer is needed to generate the boxes. We regenerate all boxes and masks from only the image and the question, with the answer removed from the prompt. Performance drops only slightly (e.g., +BBox+Mask: 71.82 → 70.46 on spatial perception), retaining 83–91% of the original gains across all models and metrics.
  • Localization is not the answer. The subquestions ask for relations, directions and truth values. Boxes only tell the model where the objects are; the model must still compute the relation.
  • Less recognition demand helps more than more location. +BBox fully specifies object locations and gives +4.5 on spatial perception. +Shape adds no location and only removes the need to recognize objects, yet gives +7.1. +Shape exceeds +BBox in all 8 model–function–metric comparisons in Table 3, which the leak account cannot explain.
  • Not every overlay helps. +Mask alone lowers spatial accuracy (61.37 → 59.70), so the gain is not a generic effect of marking the image.

Random boxes. We ran this control: boxes of the same number and size at random positions give 60.52 / 62.48 (spatial / relational), compared with 61.37 / 63.25 without boxes and 65.84 / 64.95 with Gemini boxes. Random boxes provide wrong location information, so the drop is expected under both the "mechanism" and the "information leak" accounts and cannot separate them. The comparison that does separate them is +Shape vs. +BBox above.

Oracle boxes. Ground-truth boxes are not available for CogVSR. The boxes in SHA localize the objects and parts named in each subquestion (e.g., "the dog's head" in Table 1), which are produced by our decomposition and are not annotated in any of the source benchmarks. The source annotations are also incomplete or absent: [VSR inherits COCO boxes only for whole objects in 80 categories], and SpatialEval's maze and grid questions contain no object-level annotations at all. Since Gemini boxes serve as an estimate of oracle boxes, oracle boxes would give an upper bound on the same cue and would not change the interpretation. Regarding Fig. 4 in the paper, the importance maps under different cues are highly correlated (r = 0.69–0.84, Table 10), suggesting that the activated heads do not depend on the particular cue.

Q. Strength and significance of the positive intervention.

  • Consistency. All 12 held-out cells improve (3 models × 4 benchmarks; one-sided sign test, p = 2.4×10⁻⁴), as do 32 of all 36 cells (p ≈ 10⁻⁶). Cells sharing a model are not fully independent, so we treat this as a consistency check.
  • Dose–response with sign reversal. In Table 11, every positive α (0.05–0.3) raises both spatial perception and relational reasoning, every negative α lowers both, and the drop grows with |α|. Noise would not produce such a sign-symmetric, ordered pattern. The effect peaks at α = 0.1 and remains positive up to 0.3.
  • Necessity. Masking the same heads removes 58–73% of benchmark accuracy (Table 4).

Paired tests. Pooling the four benchmarks (n = 800 paired samples per model), the gains are positive for all three models but not individually significant:

ModelΔMcNemar p95% CI (paired bootstrap)
Qwen7B+0.750.23[−0.46, 1.94]
Llama11B+0.530.38[−0.67, 1.73]
Intern8B+0.680.27[−0.52, 1.87]

We would like to highlight that the intervention is a single training-free linear shift on a small set of heads, with one α = 0.1 shared across all models and benchmarks and never tuned on any evaluation benchmark. It is designed to test direction, not to maximize accuracy. Even so, it consistently improves spatial reasoning across models and benchmarks, showing that the identified heads are not only correlated with spatial reasoning but also serve as a practical lever for improving it. This opens directions for future work, such as head-targeted fine-tuning, input-adaptive steering strengths, and combining head steering with training-based methods to obtain larger gains. We will tone down the claim accordingly in the revised version.

Q. Correlational core claim (Sec. 6.3). "Spatial heads are scarce, hence VLMs fail at spatial reasoning" is never causally tested (e.g., does fine-tuning increase their count, and does that improve reasoning?).

We respectfully note that the paper does not make this causal claim. Section 6.3 states that the limited presence of spatial heads "may be associated with weaker performance on spatial reasoning, suggesting a possible imbalance."

What the paper does establish causally is narrower: the spatial heads that exist are necessary for spatial reasoning, as masking them removes 58–73% of benchmark accuracy (Table 4), and steering them shifts accuracy in the corresponding direction (Tables 5 and 11). Whether spatial fine-tuning increases the number of such heads, and whether this in turn improves reasoning, is an interesting question. Answering it requires training and comparing multiple checkpoints, which goes beyond the scope of our analysis of frozen models. We will discuss it as future work.

Q. Inter-annotator agreement and multi-label conflicts.

The verification protocol is described in Appendix D.

  • Stage 1 (subquestion validity). Three annotators independently judge each subquestion. A subquestion is invalid if at least two annotators reject it. A decomposition is discarded if more than 40% of its subquestions are invalid or if any annotator accepts fewer than 60% of them. This removes 28.7% of the question–answer pairs.
  • Stage 2 (function labels). The three annotators independently verify each function label, and disagreements are resolved by joint discussion until consensus. Multi-label conflicts: a second label is added only when all three annotators agree that the subquestion cannot be answered without that function.

Since there are three annotators, we report Fleiss' κ along with the mean pairwise Cohen's κ. Both stages show substantial agreement (Landis & Koch, 1977):

StageJudgmentFleiss' κMean pairwise Cohen's κ
Stage 1Subquestion validity0.720.73
Stage 2Function-label correctness (before adjudication)0.680.69

Q5. Meaning and stability of the 0.001 importance threshold.

The ranked importance curves in Fig. 14 show that head importance is sparse: for all eight functions, importance drops steeply over a small fraction of heads and then flattens into a long tail of low-importance heads. The threshold 0.001 is consistent with our elbow method (Section 6.2), marking the transition from the few high-importance heads to the flat tail. It is also stable across functions: the elbow points of all eight functions lie in a narrow range around 1–2 × 10⁻³, so a single threshold reflects the same transition for every function. The threshold is used only in Section 6.1 as a descriptive cut-off for reporting sparsity and does not affect any intervention experiment.

Reviewer_BscrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether attention heads in vision-language models play different roles in spatial reasoning. The authors introduce CogVSR, which breaks spatial reasoning questions into subquestions labeled with eight cognitive functions. They then train a probe on head outputs and use gradient-based attribution to select heads associated with each function. Experiments on InternVL, Qwen2.5-VL, and Llama-Vision models suggest that relatively few heads have a large effect on task performance. The paper also studies input-based spatial cues and evaluates the selected heads through ablation and activation steering.

Critical strengths

S1. Structured task decomposition and broad model coverage.

The CogVSR decomposition makes it possible to examine individual steps in spatial reasoning rather than only final-answer accuracy. The study also covers several VLM families and model sizes.

S2. Intervention-based evaluation.

The authors go beyond probing accuracy and test the selected heads with interventions. Masking these heads usually hurts performance more than masking the same number of random heads, which supports their relevance to the tasks.

Critical weaknesses

W1. The evidence for functional specialization is still limited.

CogVSR labels describe what a subquestion is expected to require, not necessarily what the VLM does internally. The head outputs used by the probe may contain cues from the question or generated answer, even when the heads are not carrying out the proposed spatial computation. The ablation results show that the selected heads matter, but it is still unclear whether they are specific to the assigned functions. For example, the paper does not test whether removing spatial heads affects spatial tasks more than other tasks after accounting for general head importance. Cross-function ablations and controlled spatial counterfactuals would help make this case.

W2. SHA may benefit from answer-guided localization.

The SHA prompt in Appendix I includes the answer when generating boxes and masks. This allows the highlighted regions to be chosen based on the answer itself, which is not part of a standard question-answering setup. It is therefore hard to tell how much of the gain in Table 3 comes from better spatial reasoning and how much comes from answer-guided target selection. A baseline with boxes and masks generated from only the image and question would help resolve this.

W3. The out-of-domain tests use the same benchmark sources as CogVSR.

CogVSR is built from VSR, SpatialEval, 3DSRBench, and Spatial457 (Section 4.2), but Section 6.5 uses the same four benchmarks for the 'out-of-domain' tests. It is not clear whether the evaluation questions overlap with any CogVSR examples. Even if they are disjoint, this is still evaluation on held-out samples from the source benchmarks, not transfer to a new benchmark distribution. The authors should clarify the split and test on at least one benchmark that was not used to build CogVSR.

W4. The multi-label probing setup and dataset statistics need clarification.

Section 4.3 says a subquestion can have more than one cognitive label, but Appendix H describes a multi-class MLP trained with cross-entropy loss. Table 8 reports a single accuracy without explaining how the multi-label targets were handled. Also, the eight category counts in Table 6 add up to 3,755 rather than the stated 3,759. These may be reporting issues, but the labels are central to the probe, so the authors should clarify the target format, loss, evaluation metric, and dataset counts.

Minor. Please clarify the data split used for head selection.

Figure 2 reports head-importance scores on the CogVSR test set, while Section 6.2 also evaluates head ablations on that set. Please clarify which splits were used to train the probe, rank heads, choose the elbow threshold, and tune the interventions. If test examples were used to select the heads, the ablation results should also be reported on an independent split.

Questions for authors

Q1. Do spatial-head ablations affect spatial tasks more than other cognitive tasks? How do similarly important non-spatial heads compare?

Q2. Appendix I uses the answer to guide box and mask generation. How much of the improvement in Table 3 remains if localization uses only the image and question?

Q3. Are the Section 6.5 evaluation examples disjoint from the CogVSR data used for probing and head selection? Could the authors also test a benchmark from a new source?

Q4. How are multi-label targets encoded, and what loss and accuracy metric are used in Table 8? Can the Table 6 counts be reconciled?

2 comments

Thank you for your constructive comments and suggestions. Please find our point-by-point responses below.

W1. The evidence for functional specialization is still limited.

  1. Labels describe demand, by design. This paper asks whether VLMs contain heads that are associated with that demand (capacity). It does not ask whether a model spontaneously follows the same decomposition when it reasons freely (utilization); Section 7 states this difference. The labels are therefore operational definitions of the task. Whether a head is internally involved is tested by intervention and is not inferred from the label.

  2. We validate the functional relevance of the identified heads through extensive intervention experiments, rather than inferring it solely from probe results.

  3. Cross-function experiments further demonstrate the functional specialization of the identified heads. To test whether the identified heads are function-specific, we ran cross-function ablations on Qwen2.5-VL-7B. We ablated the heads of each function and evaluated performance on all eight functions (Figure 1: cross-function ablation). Ablating a function's own heads causes the largest drop in every row (8/8) compared to ablating other functional heads. For example, ablating the spatial heads drops spatial performance to 0.577, while ablating any other function's heads leaves it between 0.695 and 0.754. This specificity is not explained by general head importance.

W2. SHA may benefit from answer-guided localization.

First, we clarify that The evaluated model never sees the answer. Boxes and masks are produced by a separate model (Gemini 2.5-Flash), which uses the {answer} field only to list the object names to localize (Appendix I), while the evaluated VLM receives only the edited image and the question. And, the overlay does not contain the answer to a spatial question. A box gives only an object's name and position, whereas the spatial subquestions (Table 3) ask for a relation, direction or truth value that the model must still compute (e.g., boxes on the dog and the horse in Table 1 do not reveal whether the dog faces the horse).

To further address the concern, we regenerate all boxes and masks from only the image and the question, with the answer removed from the SHA prompt. +BBox+Mask still yields consistent gains over the original input: spatial accuracy improves by +10.30 on Intern2B and +5.13 on Llama90B, and relational accuracy by +7.58 and +4.77, respectively. Compared with the answer-guided setting, performance drops only slightly, and the question-only setting retains 83%--91% of the original gains across all metrics. +BBox+Mask also remains the best variant in all 8 settings. These results confirm that the improvement mainly comes from better spatial grounding rather than answer-guided target selection.

Table. Effect of spatial input augmentation with boxes and masks generated from the image and question only (no answer in the SHA prompt).

ModelInputSpatial llmSpatial accRelational llmRelational acc
Intern2BOriginal61.3756.8263.2558.13
+BBox65.1258.3164.2061.48
+Mask59.2459.1066.0355.92
+Color64.3159.6265.0460.57
+Shape67.4263.8567.9164.63
+Color+Shape68.3365.1868.1265.04
+BBox+Mask70.4667.1268.9465.71
Llama90BOriginal86.2182.0788.0383.92
+BBox89.6386.3489.4185.22
+Mask87.1883.0290.7384.61
+Color88.5784.4189.3085.18
+Shape90.9486.5891.2287.83
+Color+Shape91.4786.7991.5688.12
+BBox+Mask92.3187.2091.8888.69

W3. The out-of-domain tests use the same benchmark sources as CogVSR.

The question sets are disjoint. The evaluation questions per benchmark are sampled from questions not among the 400 used to build CogVSR (disjoint by question ID). At the image level, 0–24% of the evaluation images also appear in CogVSR. Removing these questions changes results by at most 0.61 points, so the intervention effects are not driven by shared images. We agree that this is evaluation on held-out samples rather than transfer to a new distribution, and we will rename this setting "held-out" and clarify the split in Section 6.5.

New benchmark not used to build CogVSR. We further evaluate on EmbSpatial-Bench, an embodied spatial reasoning benchmark built from egocentric 3D scenes, whose image sources and question formats differ from those of CogVSR.

ModelNegative (before→after)Positive (before→after)
Intern8B53.60→19.8353.60→54.30
Qwen7B51.85→18.1551.85→52.70
Llama11B47.30→15.1447.30→47.75
  • Negative intervention: Masking spatial heads lowers accuracy by 32–34 points on all three models, comparable to the drops on the four source benchmarks.
  • Positive intervention: Steering spatial perception and relational reasoning heads improves all three models.

These results show that the identified spatial heads generalize to a benchmark distribution not used to build CogVSR.

W4. The multi-label probing setup and dataset statistics need clarification.

Thank you for catching these. They are reporting errors in the appendix and do not affect the results.

Each subquestion has an 8-dimensional multi-hot target (Section 4.3, Table 1). The probe has eight sigmoid outputs and is trained with binary cross-entropy summed over the labels.

Metric. Table 8 in the paper reports exact-match accuracy: a prediction counts as correct only when the predicted label set equals the annotated label set. The following table reports both F1 scores for all six models to Table 8.

Table. Test performance (%) of the probe on different VLMs. Accuracy is exact-match over the full label set.

MetricInternVL3-2BInternVL3-8BQwen2.5VL-7BQwen2.5VL-3BLLaMA3.2-11BLLaMA3.2-90B
Accuracy78.7475.8882.1584.2289.6483.44
Macro-F173.976.082.483.389.884.6
Micro-F182.674.985.782.890.382.4

Counts. Table 6 in the paper assigns each subquestion to its primary label. We will clarify this in Table 6 and add the full label counts, including the 412 subquestions (11.0%) that carry more than one label.

Table. Distribution of cognitive skills.

Cognitive SkillCount
Spatial Perception1,026
High-level Visual Perception941
Relational Reasoning576
Decision Making498
Knowledge Recall272
Information Extraction and Understanding198
Math Reasoning138
Low-level Visual Perception106

Minor:

The probe is trained on the CogVSR training set. Head-importance scores, head ranking, and the elbow threshold are computed on the test set. We believe this does not inflate the reported effects:

  1. No parameters are fit on the test set for the ablation outcome. The test set is used only to score heads by probe accuracy, and the elbow is a single cut-off per function. Head selection never uses the ablation results, and the selection criterion (probe accuracy) differs from the evaluation metric (task accuracy after ablation).

  2. The cross-function control shares the same test set. All eight head sets are selected and evaluated in the same way. Any selection bias would therefore affect all cells of the cross-function matrix equally. Yet ablating a function's own heads causes the largest drop in every row (8/8).

  3. Most importantly, we conduct out-of-domain experiments. Section 6.5 applies the selected heads to held-out data never used for head selection, with both negative and positive interventions. We further add EmbSpatial-Bench, which is not used to build CogVSR at all. This shows that the effects generalize beyond the data used for selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.