W1. The core geometric operation is very similar to All-but-the-Top, and the novelty does not convince me
The core operation of CSRA is:
q_res = (I − UUᵀ)q
U consists of the leading singular vectors estimated from a set of boundary queries.
This is very similar in spirit to All-but-the-Top (ABTT), published at ICLR 2018 (the authors cite it only in the related work section). ABTT improves representation geometry by removing the common mean and several dominant PCA directions. CSRA follows essentially the same idea: identifying dominant directions in a representation space, removing their contributions, and using the remaining representation for similarity computation.
I understand that the two methods are applied to different representations and are not mathematically identical. However, I do not see a sufficiently clear methodological contribution beyond this existing line of work.
The authors also emphasize the exact common–residual decomposition of QK logits in Eq 5-7. BUT this decomposition follows directly from the properties of orthogonal projections and the linearity of the dot product. It works for any orthogonal projection matrix. I do not think the decomposition itself represents a substantial technical contribution.
There are also related works on geometric correction, such as IsoCLIP[1], showing that different spectral components can affect similarity geometry and that filtering certain directions can improve downstream performance.
[1] CVPR 2026 paper "IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment" by Simone Magistri, Dipam Goswami, Marco Mistretta, Bartłomiej Twardowski, Joost van de Weijer, Andrew D. Bagdanov.
The current paper mainly compares against Raw Attention, CP-LRP, and TAM, they focus more on attention visualization or attribution. These comparisons do not tell us whether the improvements come from the proposed method specifically, or simply from applying a standard geometric correction to QK representations.
I would like to see comparisons with at least the following methods:
- Mean Removal;
- Centered PCA Top-k Removal;
- Standard ABTT (Mean Removal + Top-k PCA Removal);
- PCA subspaces estimated from regular text queries;
- Random Orthogonal Subspace Removal.
These methods should be evaluated using the same backbone, layers, heads, rank, and localization pipeline.
W2. The necessity of using boundary tokens is no justified
The authors use the boundary token between the visual tokens and the target text to estimate the common subspace.
Their explanation is that, due to causal masking, this position cannot attend to the subsequent target text. Therefore, its query can be used to estimate directions that are independent of the target description.
I understand this motivation. However, the fact that a token cannot access the target text does not make it a uniquely suitable probe for estimating the common subspace.
Regular text tokens, or even random strings [2] , may also produce queries with stable shared directions. I am not convinced that the boundary token plays the special role suggested by the paper.
I would like the authors to compare the following alternatives:
- The original boundary token;
- A fixed ordinary text token;
- Random strings;
- Other tokens placed before the target description;
It would also be important to control the token positions, tokenization lengths, and the visual and textual context available to each probe.
I raise this concern partly because I have observed similar behavior with random strings in my own experiments on related methods. Of course, this does not necessarily mean the same observation will hold for CSRA, but I think this ablation is important.
[2] Efficient Streaming Language Models with Attention Sinks, Xiao et al.,ICLR 2024
W3. Raw Attention and CSRA+ use different localization pipelines, which may affect the main performance comparison
According to Appendix C.3.2, the Raw Attention map is constructed by:
- Applying Softmax over visual tokens separately for each attention head;
- Averaging the resulting attention probabilities across heads;
- Using the aggregated probabilities as the visual attention map.
However, CSRA+ is constructed differently:
- Computing raw QK logits using the residual queries;
- Averaging the residual logits across heads;
- Clipping negative values to zero;
- Applying maximum normalization, interpolation, and Otsu thresholding.
Raw Attention and CSRA+ do not use the same score transformation before head aggregation.
This is concerning because the improvements reported in Table 5 may come from two different sources:
- Removing the common subspace;
- Changing the way the localization map is constructed, including the use of raw logits, the aggregation order, and positive clipping.
Although the authors apply the same final normalization and Otsu thresholding procedure, this does not eliminate the differences introduced earlier in the pipeline.
I am also concerned about the positive clipping operation in CSRA+, where the averaged QK logits are directly transformed using max(score, 0).
softmax attn is invariant to adding the same constant to all visual logits, clipping raw logits at zero does not have this property.
SO the localization map produced by CSRA+ may be sensitive to the absolute offset of the logits and the choice of zero as the clipping threshold. Such changes do not necessarily correspond to changes in the model's attention preferences or semantic alignment.
I think the authors should provide a controlled comparison using the same pipeline, like:
- Raw QK logits;
- Raw QK logits + Positive Clipping;
- Raw QK logits + Softmax;
- Residual QK logits;
- Residual QK logits + Positive Clipping;
- Residual QK logits + Softmax.
The order of head aggregation should also be kept consistent.
Without these controls, it is difficult to determine how much of the improvement in Table 5 actually comes from removing the common subspace.
W4. The connection between the low-rank common space and attention sinks is not sufficiently established
The authors argue that the low-rank common subspace produces sink-like responses that interfere with target-related QK matching.
This is a reasonable hypothesis, but I do not think the current experiments provide enough evidence to establish this mechanism.
The authors estimate the common subspace using SVD on boundary queries and then report a high Pearson correlation between the common-space maps and the boundary-probe maps.
However, since the common space is estimated from those same types of boundary queries, a high correlation between their spatial responses is not particularly surprising.
This result shows that the estimated subspace can reproduce part of the response pattern of boundary queries. It does not necessarily demonstrate that these directions correspond to harmful attention sinks.
I would like to see more direct evidence, for example:
- belike, comparing how removing different subspaces affects sink responses and target localization
- or checking whether the common component consistently attends to the same irrelevant regions across different target descriptions.
W5. The semantic probing experiments in Section 3 are not sufficiently informative
Table 1 shows that some semantic category structure is preserved after query projection.
However, in Appendix Table 8, the spectrum-matched random projections achieve results very close to those of the learned query projections.
This shows that the experiments mainly show that category structure survives dimensionality reduction. They do not clearly establish that the learned query projection has a special ability to preserve target-related semantics.
also, the authors use six semantic categories: Part, Action, Shape, Material, Color, and Position.
The ability to distinguish individual terms from these broad categories is quite different from understanding complex referring expressions, where the model needs to handle combinations of attributes, object relationships, and spatial constraints.
I would therefore be cautious about using these results as evidence that the query representations preserve the semantic information needed for reliable visual grounding.
W6. The analysis of query-side and key-side geometry is incomplete
The method estimates the common subspace from boundary queries and removes its contribution from the target query.
The authors correctly point out in Eq.7 that, for the same orthogonal projection matrix, filtering the query side is equivalent to filtering the key side in the QK dot product.
My concern is slightly different: Why should the subspace responsible for misalignment be estimated entirely from the query side?
Visual keys may also contain anisotropic structures, dominant shared directions, or geometric components that are poorly matched with the query space.
A subspace estimated only from boundary queries may not capture all the directions that interfere with cross-modal alignment.
I think a more complete geometric analysis would be useful. For example:
- Estimate a common subspace U_Q from query-side statistics;
- Independently estimate a subspace U_K from visual-key statistics;
- Compare Query-derived, Key-derived, and joint correction under the same evaluation pipeline.