acceptodds
Under review as a conference paper at ICLR 2027

CSRA: Decomposing Native Visual QK Alignment in Vision–Language Models

Abstract

Visual-language understanding in vision language models (VLMs) relies on effective cross-modal alignment. However, existing analyses based on attention maps primarily reveal where models attend, offering limited insight into how alignment forms and where failures originate. We show that cross-modal alignment is directly encoded in native query-key (QK) interactions and distinguish three sources of alignment failure: loss of textual semantic information during query projection, insufficient discriminability of visual target keys, and incorrect matching between queries and keys.Further analysis shows that even when queries preserve semantic structure and keys support target discrimination, their interactions can be dominated by a low-rank shared subspace associated with attention sinks, leading to deviations from semantic alignment. Based on this finding, we propose Common-Space Residual Attribution (CSRA), using singular value decomposition (SVD) to identify and remove the projection components responsible for this interference in QK interactions, thereby suppressing the influence of the sink subspace.Experiments on Qwen2.5-VL, LLaVA-1.5, and InternVL3 demonstrate improved target localization, with RefCOCO mIoU gains of up to 27.1 % over native attention. The estimated subspace also exhibits cross-dataset transferability. Our findings reveal how QK interactions shape cross-modal alignment and provide a unified framework for diagnosing potential alignment failures and guiding mechanism-driven alignment enhancement.

Primary area:
interpretability and explainable AI
Keywords:
Vision–Language Models, Attention Sinks, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jdCkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

CSRA removes a low-rank subspace estimated from boundary-token queries to expose target-specific QK alignment. The paper reports substantial localization gains across three VLMs, though some central mechanistic claims need stronger controls.

Critical strengths

Estimating a reusable subspace from boundary queries is an interesting distinction from TAM and VAR.

Localization improvements are substantial

The fixed-key decomposition is clean, and the analysis includes cases where suppressing the shared response does not recover good localization.

Critical weaknesses
  • The main comparison does not isolate subspace removal. Averaging logits instead of probabilities can itself change the localization map and its thresholded mask. (Appendix C.3.2 defines Raw as averaged softmax probabilities, whereas CSRA+ averages residual logits before clipping.)

  • The evidence does not establish that the estimated subspace specifically captures sink interference rather than generic shared structure. Being independent of the referring expression does not make a component irrelevant to localization. (Section 4.1 fits the basis from image-conditioned boundary queries; Table 4 measures similarity to that probe, while Table 6 tests rank without alternative-subspace controls.)

  • Generalization to unseen images is not cleanly established. (Appendix C.1 explicitly allows the same image across fitting and evaluation; Appendix C.3.1 does not establish image disjointness for the COCO transfer experiment.)

Questions for authors

How much improvement remains when raw and residual QK scores use identical aggregation and postprocessing?

Does the boundary-derived basis outperform random subspaces, a mean-query direction, and principal components of ordinary queries?

Reviewer_XHBmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-modal alignment in VLMs through their native Query–Key interactions. It tries to explain why attention maps sometimes fail to localize the referred visual regions, even when the internal representations still contain semantic information. I find the research question interesting, and the results show that removing certain low-rank components can improve spatial localization. BUT I have several concerns about the novelty of the method, the motivation behind the boundary-token design, the interpretation of attention sink.

Critical strengths

S1. Simple method

CSRA uses the queries and keys already available in pretrained VLMs. It estimates a low-rank common subspace through SVD during calibration and removes the corresponding projection when constructing the attribution maps. it does not require any additional trainable modules or parameter updates

S2. Clear localization improvements across multiple models

The authors evaluate CSRA on three different VLM backbones. In Table 5, CSRA+ outperforms raw attention in all nine RefCOCO settings. The improvements are quite substantial at some intermediate layers, showing that certain dominant directions in the query space may indeed affect the quality of the localization maps

Critical weaknesses

W1. The core geometric operation is very similar to All-but-the-Top, and the novelty does not convince me

The core operation of CSRA is:

q_res = (I − UUᵀ)q U consists of the leading singular vectors estimated from a set of boundary queries.

This is very similar in spirit to All-but-the-Top (ABTT), published at ICLR 2018 (the authors cite it only in the related work section). ABTT improves representation geometry by removing the common mean and several dominant PCA directions. CSRA follows essentially the same idea: identifying dominant directions in a representation space, removing their contributions, and using the remaining representation for similarity computation.

I understand that the two methods are applied to different representations and are not mathematically identical. However, I do not see a sufficiently clear methodological contribution beyond this existing line of work.

The authors also emphasize the exact common–residual decomposition of QK logits in Eq 5-7. BUT this decomposition follows directly from the properties of orthogonal projections and the linearity of the dot product. It works for any orthogonal projection matrix. I do not think the decomposition itself represents a substantial technical contribution.

There are also related works on geometric correction, such as IsoCLIP[1], showing that different spectral components can affect similarity geometry and that filtering certain directions can improve downstream performance. [1] CVPR 2026 paper "IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment" by Simone Magistri, Dipam Goswami, Marco Mistretta, Bartłomiej Twardowski, Joost van de Weijer, Andrew D. Bagdanov.

The current paper mainly compares against Raw Attention, CP-LRP, and TAM, they focus more on attention visualization or attribution. These comparisons do not tell us whether the improvements come from the proposed method specifically, or simply from applying a standard geometric correction to QK representations.

I would like to see comparisons with at least the following methods:

  • Mean Removal;
  • Centered PCA Top-k Removal;
  • Standard ABTT (Mean Removal + Top-k PCA Removal);
  • PCA subspaces estimated from regular text queries;
  • Random Orthogonal Subspace Removal. These methods should be evaluated using the same backbone, layers, heads, rank, and localization pipeline.

W2. The necessity of using boundary tokens is no justified

The authors use the boundary token between the visual tokens and the target text to estimate the common subspace. Their explanation is that, due to causal masking, this position cannot attend to the subsequent target text. Therefore, its query can be used to estimate directions that are independent of the target description.

I understand this motivation. However, the fact that a token cannot access the target text does not make it a uniquely suitable probe for estimating the common subspace.

Regular text tokens, or even random strings [2] , may also produce queries with stable shared directions. I am not convinced that the boundary token plays the special role suggested by the paper.

I would like the authors to compare the following alternatives:

  1. The original boundary token;
  2. A fixed ordinary text token;
  3. Random strings;
  4. Other tokens placed before the target description;

It would also be important to control the token positions, tokenization lengths, and the visual and textual context available to each probe. I raise this concern partly because I have observed similar behavior with random strings in my own experiments on related methods. Of course, this does not necessarily mean the same observation will hold for CSRA, but I think this ablation is important.

[2] Efficient Streaming Language Models with Attention Sinks, Xiao et al.,ICLR 2024

W3. Raw Attention and CSRA+ use different localization pipelines, which may affect the main performance comparison

According to Appendix C.3.2, the Raw Attention map is constructed by:

  1. Applying Softmax over visual tokens separately for each attention head;
  2. Averaging the resulting attention probabilities across heads;
  3. Using the aggregated probabilities as the visual attention map.

However, CSRA+ is constructed differently:

  1. Computing raw QK logits using the residual queries;
  2. Averaging the residual logits across heads;
  3. Clipping negative values to zero;
  4. Applying maximum normalization, interpolation, and Otsu thresholding.

Raw Attention and CSRA+ do not use the same score transformation before head aggregation. This is concerning because the improvements reported in Table 5 may come from two different sources:

  1. Removing the common subspace;
  2. Changing the way the localization map is constructed, including the use of raw logits, the aggregation order, and positive clipping.

Although the authors apply the same final normalization and Otsu thresholding procedure, this does not eliminate the differences introduced earlier in the pipeline.

I am also concerned about the positive clipping operation in CSRA+, where the averaged QK logits are directly transformed using max(score, 0). softmax attn is invariant to adding the same constant to all visual logits, clipping raw logits at zero does not have this property.

SO the localization map produced by CSRA+ may be sensitive to the absolute offset of the logits and the choice of zero as the clipping threshold. Such changes do not necessarily correspond to changes in the model's attention preferences or semantic alignment.

I think the authors should provide a controlled comparison using the same pipeline, like:

  • Raw QK logits;
  • Raw QK logits + Positive Clipping;
  • Raw QK logits + Softmax;
  • Residual QK logits;
  • Residual QK logits + Positive Clipping;
  • Residual QK logits + Softmax.

The order of head aggregation should also be kept consistent.

Without these controls, it is difficult to determine how much of the improvement in Table 5 actually comes from removing the common subspace.

W4. The connection between the low-rank common space and attention sinks is not sufficiently established

The authors argue that the low-rank common subspace produces sink-like responses that interfere with target-related QK matching. This is a reasonable hypothesis, but I do not think the current experiments provide enough evidence to establish this mechanism.

The authors estimate the common subspace using SVD on boundary queries and then report a high Pearson correlation between the common-space maps and the boundary-probe maps. However, since the common space is estimated from those same types of boundary queries, a high correlation between their spatial responses is not particularly surprising.

This result shows that the estimated subspace can reproduce part of the response pattern of boundary queries. It does not necessarily demonstrate that these directions correspond to harmful attention sinks.

I would like to see more direct evidence, for example:

  1. belike, comparing how removing different subspaces affects sink responses and target localization
  2. or checking whether the common component consistently attends to the same irrelevant regions across different target descriptions.

W5. The semantic probing experiments in Section 3 are not sufficiently informative

Table 1 shows that some semantic category structure is preserved after query projection. However, in Appendix Table 8, the spectrum-matched random projections achieve results very close to those of the learned query projections.

This shows that the experiments mainly show that category structure survives dimensionality reduction. They do not clearly establish that the learned query projection has a special ability to preserve target-related semantics.

also, the authors use six semantic categories: Part, Action, Shape, Material, Color, and Position.

The ability to distinguish individual terms from these broad categories is quite different from understanding complex referring expressions, where the model needs to handle combinations of attributes, object relationships, and spatial constraints.

I would therefore be cautious about using these results as evidence that the query representations preserve the semantic information needed for reliable visual grounding.

W6. The analysis of query-side and key-side geometry is incomplete

The method estimates the common subspace from boundary queries and removes its contribution from the target query.

The authors correctly point out in Eq.7 that, for the same orthogonal projection matrix, filtering the query side is equivalent to filtering the key side in the QK dot product.

My concern is slightly different: Why should the subspace responsible for misalignment be estimated entirely from the query side?

Visual keys may also contain anisotropic structures, dominant shared directions, or geometric components that are poorly matched with the query space.

A subspace estimated only from boundary queries may not capture all the directions that interfere with cross-modal alignment.

I think a more complete geometric analysis would be useful. For example:

  1. Estimate a common subspace U_Q from query-side statistics;
  2. Independently estimate a subspace U_K from visual-key statistics;
  3. Compare Query-derived, Key-derived, and joint correction under the same evaluation pipeline.
Questions for authors
  1. What does CSRA offer beyond standard ABTT or PCA-based component removal, either algorithmically or in terms of the underlying mechanism? (You can answer me by fixing W1)

  2. Whether the boundary token has a unique computational role, or whether it is simply a convenient way to obtain queries that do not depend on the subsequent target text.

  3. Can the authors provide stronger evidence that the identified low-rank subspace is actually responsible for the observed misalignment?

Reviewer_tHduOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes native visual query-key alignment in vision-language models and separates a common low-rank component from residual query contributions. CSRA uses the residual to create localization maps and evaluates them on RefCOCO and a RefCOCO-to-COCO transfer setting across three VLMs.

Critical strengths
  1. The query/key diagnosis makes a useful distinction between semantic information being present and its use in attention-based localization.

  2. Table 5 shows CSRA+ above raw attention in all nine RefCOCO settings, including Qwen2.5-VL layer 19 at 36.97 versus 20.86 mIoU; the frozen basis also transfers to COCO.

Critical weaknesses
  1. The demonstrated gain is primarily attribution-map mIoU. Without a causal intervention or another behavioral faithfulness test, the analysis does not yet establish whether the diagnosed Q/K component explains the VLM's decisions.

  2. Performance is layer-dependent relative to other explanations: at LLaVA-1.5 layer 31, CSRA+ scores 3.80 versus CP-LRP 7.46 and TAM 6.97. The layer-selection rule and these failure modes need clarification.

Questions for authors
  1. Can an intervention on the identified subspace or CSRA-highlighted regions establish a connection to the VLM's decisions, beyond localization-map overlap?

  2. What label-free rule selects layer and rank, and why does the deep LLaVA setting underperform CP-LRP and TAM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.