- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies spatial biases in VLM feature recognition using controlled synthetic images. The authors evaluate atomic and compositional features, varying their position, fragmentation, scale, and distractors. They find that recognition varies across spatial positions even for patch-aligned features, fragmentation is often harmless, upsampling improves recognition, and longer visual sequences and distractors degrade performance.
Critical strengths
Controlled experimental setup. The paper isolates spatial location from patch fragmentation, showing that feature recognition can vary even when features remain aligned with the encoder patch grid. The experiments cover several feature types, models, and visual conditions. The finding that fragmentation is not consistently harmful, while upsampling often helps, is interesting.
Critical weaknesses
Tokenization ambiguity. The paper treats encoder patches as visual tokens, but models such as Qwen and InternVL merge multiple encoder patches into downstream visual tokens. Thus, fragmentation across encoder patches does not necessarily imply fragmentation across LLM-facing tokens, undermining the interpretation of Sections 4.2–4.3. Thus, it is unclear whether their experiments properly isolate the effects they claim to study.
Limited statistical validation of spatial anisotropy.
Some tasks use very few stimuli per spatial position. For example, color recognition uses only six examples, so a single mistake changes accuracy by 16.7 percentage points. Yet the paper interprets detailed spatial patterns without adequately quantifying their reliability. Figure 3 also normalizes each heatmap by its own spatial standard deviation, making it difficult to compare the actual magnitude of positional differences across features. Also, Appendix C.2 shows that rephrasing prompts can substantially change both accuracy and spatial patterns. Overall, it is unclear how robust and generalizable the reported anisotropy patterns are.
Questions for authors
Can you provide statistical analysis and uncertainty estimates for positional differences, accounting for the small number of stimuli per position (e.g., Color)?
Are the patch grids defined before or after visual-token merging? Do the fragmentation results hold when evaluated relative to the actual LLM-facing token boundaries?
2 comments
Thank you for the detailed response and additional experiments. I appreciate the clarified discussion of prompt sensitivity and the limitations of the heatmap normalization. I believe the authors have addressed my original concerns. I am updating my score to weak accept.
Dear Reviewer zCry, we thank you for your insightful review and the valuable feedback. We are happy to hear that you found our results "interesting" and our paper "clear and easy to follow". Please find a detailed answer to your comments below.
We apologize for the confusion. We have updated the paper to clarify that we study patch grids as seen by the encoder, not the vision tokens of an LLM.
We ran a full-grid fragmentation sweep for Qwen that separates fragmentation within one merged LLM token from fragmentation across two or four. Crossing LLM-token boundaries has no meaningful effect (2 tokens covered: -0.08 pp, 95% CI [-0.34, 0.17]; 4 tokens covered after position control: ≤0.4 pp, not significant), and feature-level trends are unchanged. Our fragmentation findings therefore hold at the LLM-token level. We added this analysis to the new Appendix §C.
We would like to correct that only the color task uses few samples. All others use 54 to 150 samples per position (Tab. 3). The reason for the limited number of color samples is 1) VLMs are poor at naming colors (e.g., see https://arxiv.org/abs/2504.10514), so we restricted the set to unambiguous colors, and 2) there is no clean way to probe atomic color recognition without attribute binding.
In any case, our paper is more concerned with the fact that anisotropy exists, rather than the actual magnitude. Even a single error under translation is therefore of interest, considering that responses are generated with greedy decoding.
We now state this in the caption and report magnitudes and their uncertainty in a new appendix (Appendix §B, Tab. 2).
Decoding is greedy, so each cell is a deterministic observation: when, e.g., LLaVA misnames white at a corner but names it correctly one patch away, that is a property of the model, not sampling noise. To show how much a position's accuracy rate depends on the specific stimuli, we compute stimulus-bootstrap CIs. After subtracting the variance expected from stimulus sampling, the positional SD reaches 11.2 pp [9.7, 12.5] (Qwen Color-Position), and 9 of 21 maps have CIs excluding zero. Regional effects are large: Shape in LLaVA's first row is -32 pp [-44, -21], and Size at Qwen's left edge is -13 pp [-25, -2]. We swapped out a poorly supported observation in Sec. 4.1 (LLaVA color revival). Thank you for bringing this to our attention! If you can point us to further specific text passages that need rephrasing, we are happy to update them.
The per-heatmap scaling was chosen on purpose. Our claim in §4.1 is about the existence and structure of positional variation, and a shared scale would make low-variance features look uniform. We agree that this hides the absolute magnitude, so that, e.g., LLaVA's size map (std = 0.2 pp) looks as pronounced as Qwen's Color-Position map (std = 11.5 pp). Therefore, the spatial mean ± std is printed under each panel and in Table 1.
Thank you for bringing this to our attention. We updated the paper to be more cautious about the observed heatmaps being prompt-specific. We would like to emphasize that this does not interfere with our study: we are not claiming that the same patterns will emerge across all prompts, but rather that all kinds of prompts will exhibit some anisotropy. We still see a high correlation between anisotropy patterns across reformulated patterns on a global scale, as can be seen in the new Appendix Tab. 3. While some features show differences in their spatial patterns, the salient patterns described in 4.1 still hold.
Once again, we thank you for your valuable feedback! We look forward to further discussing this if there are remaining questions, and thank you for your careful consideration of our paper.