acceptodds
Under review as a conference paper at ICLR 2027

Same same, but different: VLMs Do Not Recognize Features Equally Anywhere in an Image

Abstract

Vision-Language Models (VLMs) have achieved impressive performance on a wide range of tasks requiring visual understanding. Despite this progress, their predictions remain surprisingly brittle and sensitive to small spatial variations. Using synthetic experiments that isolate specific visual factors, we systematically study anisotropic feature recognition in VLMs. We find that modern VLMs reliably recognize simple atomic features contained within single patches, but continue to struggle with more complex atomic and compositional features. Crucially, the same feature can be recognized with considerably different fidelity depending on its position in the input image, with spatial anisotropy patterns that vary across feature types and models. We further show that fragmentation across patches is rarely harmful, while upsampling a feature over more patches substantially improves recognition, particularly for compositional features. Conversely, visual distractors and longer vision-token sequences degrade feature recognition and introduce additional positional biases. Taken together, our results reveal that feature recognition in VLMs remains strongly anisotropic and sensitive to the allocation of visual information across tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision representation, robustness, multimodal LLMs, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zCryOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies spatial biases in VLM feature recognition using controlled synthetic images. The authors evaluate atomic and compositional features, varying their position, fragmentation, scale, and distractors. They find that recognition varies across spatial positions even for patch-aligned features, fragmentation is often harmless, upsampling improves recognition, and longer visual sequences and distractors degrade performance.

Critical strengths

Controlled experimental setup. The paper isolates spatial location from patch fragmentation, showing that feature recognition can vary even when features remain aligned with the encoder patch grid. The experiments cover several feature types, models, and visual conditions. The finding that fragmentation is not consistently harmful, while upsampling often helps, is interesting.

Critical weaknesses

Tokenization ambiguity. The paper treats encoder patches as visual tokens, but models such as Qwen and InternVL merge multiple encoder patches into downstream visual tokens. Thus, fragmentation across encoder patches does not necessarily imply fragmentation across LLM-facing tokens, undermining the interpretation of Sections 4.2–4.3. Thus, it is unclear whether their experiments properly isolate the effects they claim to study.

Limited statistical validation of spatial anisotropy.

Some tasks use very few stimuli per spatial position. For example, color recognition uses only six examples, so a single mistake changes accuracy by 16.7 percentage points. Yet the paper interprets detailed spatial patterns without adequately quantifying their reliability. Figure 3 also normalizes each heatmap by its own spatial standard deviation, making it difficult to compare the actual magnitude of positional differences across features. Also, Appendix C.2 shows that rephrasing prompts can substantially change both accuracy and spatial patterns. Overall, it is unclear how robust and generalizable the reported anisotropy patterns are.

Questions for authors

Can you provide statistical analysis and uncertainty estimates for positional differences, accounting for the small number of stimuli per position (e.g., Color)?

Are the patch grids defined before or after visual-token merging? Do the fragmentation results hold when evaluated relative to the actual LLM-facing token boundaries?

2 comments

Dear Reviewer zCry, we thank you for your insightful review and the valuable feedback. We are happy to hear that you found our results "interesting" and our paper "clear and easy to follow". Please find a detailed answer to your comments below.

Are the patch grids defined before or after visual-token merging? Do results hold at LLM-facing token boundaries?

We apologize for the confusion. We have updated the paper to clarify that we study patch grids as seen by the encoder, not the vision tokens of an LLM.

We ran a full-grid fragmentation sweep for Qwen that separates fragmentation within one merged LLM token from fragmentation across two or four. Crossing LLM-token boundaries has no meaningful effect (2 tokens covered: -0.08 pp, 95% CI [-0.34, 0.17]; 4 tokens covered after position control: ≤0.4 pp, not significant), and feature-level trends are unchanged. Our fragmentation findings therefore hold at the LLM-token level. We added this analysis to the new Appendix §C.

Some tasks use very few stimuli per spatial position.

We would like to correct that only the color task uses few samples. All others use 54 to 150 samples per position (Tab. 3). The reason for the limited number of color samples is 1) VLMs are poor at naming colors (e.g., see https://arxiv.org/abs/2504.10514), so we restricted the set to unambiguous colors, and 2) there is no clean way to probe atomic color recognition without attribute binding.

In any case, our paper is more concerned with the fact that anisotropy exists, rather than the actual magnitude. Even a single error under translation is therefore of interest, considering that responses are generated with greedy decoding.

Provide statistical analysis and uncertainty estimates for positional differences.

We now state this in the caption and report magnitudes and their uncertainty in a new appendix (Appendix §B, Tab. 2).

Decoding is greedy, so each cell is a deterministic observation: when, e.g., LLaVA misnames white at a corner but names it correctly one patch away, that is a property of the model, not sampling noise. To show how much a position's accuracy rate depends on the specific stimuli, we compute stimulus-bootstrap CIs. After subtracting the variance expected from stimulus sampling, the positional SD reaches 11.2 pp [9.7, 12.5] (Qwen Color-Position), and 9 of 21 maps have CIs excluding zero. Regional effects are large: Shape in LLaVA's first row is -32 pp [-44, -21], and Size at Qwen's left edge is -13 pp [-25, -2]. We swapped out a poorly supported observation in Sec. 4.1 (LLaVA color revival). Thank you for bringing this to our attention! If you can point us to further specific text passages that need rephrasing, we are happy to update them.

Figure 3 also normalizes each heatmap by its own spatial standard deviation

The per-heatmap scaling was chosen on purpose. Our claim in §4.1 is about the existence and structure of positional variation, and a shared scale would make low-variance features look uniform. We agree that this hides the absolute magnitude, so that, e.g., LLaVA's size map (std = 0.2 pp) looks as pronounced as Qwen's Color-Position map (std = 11.5 pp). Therefore, the spatial mean ± std is printed under each panel and in Table 1.

Unclear how robust and generalizable the reported anisotropy patterns are under rephrasing.

Thank you for bringing this to our attention. We updated the paper to be more cautious about the observed heatmaps being prompt-specific. We would like to emphasize that this does not interfere with our study: we are not claiming that the same patterns will emerge across all prompts, but rather that all kinds of prompts will exhibit some anisotropy. We still see a high correlation between anisotropy patterns across reformulated patterns on a global scale, as can be seen in the new Appendix Tab. 3. While some features show differences in their spatial patterns, the salient patterns described in 4.1 still hold.

Once again, we thank you for your valuable feedback! We look forward to further discussing this if there are remaining questions, and thank you for your careful consideration of our paper.

Thank you for the detailed response and additional experiments. I appreciate the clarified discussion of prompt sensitivity and the limitations of the heatmap normalization. I believe the authors have addressed my original concerns. I am updating my score to weak accept.

Reviewer_TgiEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper asks whether a VLM recognizes the same visual feature equally well regardless of where it appears in the image. To isolate this from other confounds, the authors build a synthetic testbed where a single patch containing one feature is placed on a black canvas so it occupies exactly one vision token, and a multiple-choice VQA prompt then probes it. Four atomic features (shape, color, orientation, size) and three binding tasks (shape-color, position-color, color-position) are tested on three open VLMs (Qwen, InternVL, LLaVA) along five axes: the patch’s position on the grid (every cell), fragmentation of the patch across 2–4 tokens, upsampling of the patch to cover 2–8× more tokens, surrounding distractors (noise or clutter), and placement in a 121× longer token sequence. The main finding is that recognition accuracy varies substantially with position even when the feature sits entirely inside one token, so translation variance is not simply a fragmentation artifact. Simple atomic features (shape, color) survive single-token compression almost perfectly, while orientation, size, and the binding tasks degrade. Fragmentation is rarely harmful and sometimes helps slightly, upsampling improves recognition considerably (most of all for binding), and both distractors and longer sequences reduce accuracy and alter the spatial pattern. The authors conclude that feature recognition in current VLMs is strongly anisotropic and propose the testbed as a diagnostic for separating failures due to position, fragmentation, scale, distractors, and sequence length.

Critical strengths
  1. Clean factor isolation: Prior work translates whole natural images, which moves every token at once. Controlling one active token and varying position, fragmentation, scale, and context separately is the paper’s real contribution, and it’s well motivated.

  2. Upsampling helps binding; fragmentation isn’t the main bottleneck; this cuts against token-merging recommendations (Renggli et al.). I like this result.

  3. The writing is clear.

Critical weaknesses
  1. Appendix C.2 shows that a semantically equivalent rephrasing shifts Qwen’s color-position accuracy from 79.2% to 40.1% and size from 54.4% to 68.4%, with visibly different spatial maps. That means the Fig. 3 heatmaps can partly reflect one specific prompt wording. The main text should average over prompts or state which spatial findings are prompt-robust, rather than leaving this as an appendix caution.

  2. Everything is end-to-end VQA accuracy, so it cannot say whether anisotropy arises in the ViT (position embeddings, attention bias), the projector, or the LLM’s attention over the token sequence. The “start/center/end” framing and the RoPE speculation in Appendix B hint at an explanation, but nothing is tested yet.

  3. Three open models, differing simultaneously in encoder, patch size, tiling vs. native resolution, and learnable vs. RoPE embeddings, so nothing can be attributed to any one factor. Also, the Z_TV metric grows roughly with sqrt of number of grid cells, so InternVL’s 32×32 grid is expected to score about twice Qwen’s 16×16 grid for identical spatial structure.

Questions for authors
  1. Appendix C.2 shows that a rephrased prompt changes both the mean accuracy and the spatial map (e.g., color-position drops from 79.2% to 40.1%). Please report how well the heatmaps correlate across the three prompts, as done for shuffled options in Appendix C.1, and state which specific position effects (Qwen’s size collapse at the left edge, LLaVA’s first-row shape failure, the center-vs-edge pattern) survive averaging over prompts.

  2. In Section 4.2, when a patch straddles 2 or 4 tokens, which on-grid accuracy is it compared against: the mean of the overlapped cells, a single adjacent cell, or the best of them?

2 comments

Dear Reviewer TgiE, we thank you for your insightful review and the valuable feedback. We are happy to hear that you "like the results" and find our paper "well motivated" and "clear and easy to follow". Please find a detailed answer to your comments below.

Heatmaps can partly reflect one specific prompt wording/use prompt averaging

Thank you for the heads up. We agree that the patterns correlate with the prompt, but would like to emphasize that this does not interfere with our study: we are not claiming that the same patterns will emerge across all prompts, but rather that all kinds of prompts will exhibit some anisotropy. We therefore do not believe that prompt averaging would be a good strategy for our experiment. Based on your suggestion, we updated the paper to be more upfront about prompt-specific results and extended the discussion in §E.2 (previously §C.2), see also the answer below.

Please report how well the heatmaps correlate across the three prompts and state which specific position effects survive rephrasing.

We thank you very much for the suggestion to compute the correlation coefficient! We added Tab. 3 to the appendix to show correlations between all language ablations and the baseline. For Qwen, even the most divergent result (Prompt #2) remains strongly correlated at 0.86 on a global level. The weakest feature correlation (color-position) is still moderately high at 0.57. For LLaVA, the correlations are weaker, albeit with a strong global correlation (0.73).

Interestingly, all position effects described in Sec. 4.1 survive rephrasing on both models: Qwen’s size still collapses at the left edge, LLaVA’s shape still collapses in the top row, and color collapse in edges and the center. We have extended the discussion in §E.2 (former §C.2) based on your suggestions.

Everything is end-to-end VQA accuracy, so it cannot say whether anisotropy arises

We agree that our paper does not pinpoint the source of anisotropy, except for the hints mentioned by the reviewer (and these are architecture- and feature-specific). We are planning to investigate these sources in future work, but believe that a separate study (potentially by feature (group)) may be more suitable. In this work, we attempted to lay down the groundwork to disentangle various features in the input space that affect anisotropy.

Three different models, so nothing can be attributed to any one factor.

We agree that all 3 models are fundamentally different and yet anisotropy emerges in all of them, showing that anisotropy is not caused by a single architectural choice. Unfortunately, it is very hard to find model checkpoints to ablate individual factors in detail, and we would appreciate any pointers from the reviewer!

Z_TV metric is not normalized.

We apologize for the confusion! We already control the growth by the first factor, but given the concerns raised by other reviewers and the fact that we barely utilize the metric in our study, we have decided to fully remove it from the paper.

In Section 4.2, when a patch straddles 2 or 4 tokens, which on-grid accuracy is it compared against:

We apologize for not stating this more clearly: the mean on-grid accuracy over the patches they overlap, i.e., the 2x2 block of on-grid positions covered by the fragmented placements (each patch of the block is covered equally often). We have added this clarification to the paper.

Once again, we thank you for your valuable feedback! We look forward to further discussing this if there are remaining questions, and thank you for your careful consideration of our paper.

Thank you for your response and clarification. I will maintain my original positive score and increasing the the soundness to 3.

Reviewer_Bg1oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies position-dependent feature recognition in VLMs using controlled synthetic stimuli. A small target containing an atomic feature or an attribute-binding task is placed at different image locations, initially aligned with a single encoder patch. The authors evaluate seven feature families on Qwen3.6, InternVL3.5, and LLaVA1.5, and examine on-grid position, fragmentation, upsampling, distractors, and longer visual contexts. They find that recognition varies across positions even without cross-patch fragmentation; fragmentation often preserves or improves aggregate accuracy, while upsampling generally helps and distractors or larger canvases often hurt. The main contribution is a diagnostic testbed and feature-specific spatial accuracy maps, rather than a new model or robustness intervention.

Critical strengths
  1. The patch-aligned, single-target design provides a useful and interpretable diagnostic extension of prior translation-robustness studies. It establishes that cross-patch fragmentation is not necessary for position-dependent errors. The effect can be substantial: Section 4.1 reports Qwen's aspect-ratio accuracy ranging from approximately chance (33.3%) at some locations to 72.2% at the best location, with a spatial mean of 54.4%. This is a meaningful observation beyond visually different heatmaps.

  2. The separation of feature families and perturbation axes reveals task-dependent behavior that aggregate VQA accuracy can obscure. In particular, the distinction between simple atomic features and binding tasks, together with the non-aggregated results in Figures 14–17, makes the testbed useful for diagnosing failure modes. The appendix also reports prompt sensitivity and exceptions to the aggregate trends, and the limitations acknowledge that generated answers measure end-to-end feature utilization rather than directly identifying information loss.

Critical weaknesses
  1. The strongest causal framing is not established by the experiments. The conclusion states that translation variance is “not a consequence of fragmentation.” However, the on-grid result establishes that fragmentation is not necessary, not that it cannot contribute. The fragmentation experiment probes three representative locations and limited alignments, and Section 4.2 itself reports detrimental effects for shape–color binding and for Qwen's orientation recognition. Moreover, prior shift-equivariant ViT work already identifies positional encoding, attention, and patch merging alongside tokenization (Ding et al., 2023; Rojas-Gomez et al., 2024). Improved accuracy after fragmentation does not contradict a failure of shift equivariance. The related scale and context experiments also change multiple properties: upsampling changes target size and patch occupancy, while increasing canvas resolution changes positional ranges and, for InternVL, tile/thumbnail processing. These are useful behavioral results, but they do not isolate the claimed mechanisms. Since the paper's main value is explanatory, the distinction materially affects my assessment.

  2. The interpretation of the spatial statistic does not justify the cross-model anisotropy ranking. Section 3 defines Z_TV by standardizing total variation against spatial permutations. This measures spatial organization relative to a shuffled map, rather than the magnitude of position-dependent accuracy differences: multiplying all deviations in a map by a positive constant scales TV, the permutation mean, and the permutation standard deviation together, leaving Z_TV unchanged. Consequently, the statement in Section 4.1 that Qwen has the least and InternVL the greatest anisotropy, based on Figure 4, needs qualification or complementary effect-size metrics. The position-dependent errors themselves remain convincing, but the comparative claim is not supported by this statistic alone.

Questions for authors
  1. Can the authors revise or substantiate the claim that fragmentation is not a cause of translation variance? In particular, what paired fragmentation effects and uncertainty support the broader conclusion, given the limited alignments and the negative task-specific effects already reported? Clarifying the intended scope could materially improve my assessment.

  2. Does Figure 4 intend to compare spatial clustering or the magnitude of positional sensitivity? Please report a complementary magnitude measure (e.g., spatial standard deviation or percentile gap) and explain whether the cross-model ranking remains supported. Please also specify how the 95% confidence intervals in Figures 5–8 are computed, including the aggregation and resampling unit.

  3. Please clarify the actual encoder-patch and post-merger visual-token counts, including InternVL's tiles and thumbnail, for the fragmentation and long-context experiments. For the prompt controls, can the authors provide a comparison under matched batching/inference settings? Figure 12 shows Color–Position accuracy changing from 79.2% to 40.1% with a rephrased prompt under a different inference setup. These clarifications would help distinguish robust spatial effects from preprocessing and task-framing effects.

My current recommendation is weak rejection: the controlled diagnostic is useful, but the causal scope and the interpretation of the main spatial statistic need correction or additional evidence. A convincing response on these points could change my recommendation.

3 comments

Dear Reviewer Bg1o, we thank you for your insightful review and the valuable feedback. We are happy to hear that you find our diagnostic "useful", our observations "meaningful", and find our paper "well motivated" and "clear and easy to follow". Please find a detailed answer to your comments below.

Substantiate the claim that fragmentation is not a cause of translation variance.

We thank the reviewer for catching this and agree with their logic: the on-grid result shows fragmentation is not necessary for translation variance, not that it never contributes. We changed the conclusion from "not a consequence of fragmentation" to "does not require fragmentation and is not primarily caused by it," and stated the same scope in Sec. 4.2. We believe the evidence supports this narrower claim well:

All fragmentation results are paired per stimulus and prompt against the on-grid accuracy of the exact patches covered, with bootstrap CIs over stimuli. Aggregated, fragmentation is close to neutral (Qwen +2.1 pp [1.0, 3.1]; LLaVA +0.3 pp [-0.4, 1.1]; InternVL -0.4 pp). The detrimental cases the reviewer cites are real but modest: shape-color -4.3 pp (Qwen), -8.6 pp [-13.3, -4.3] (InternVL); Qwen orientation -3.0 pp [-5.2, -0.9]. However, in the same models, moving a non-fragmented feature across the grid swings accuracy from 72 % to chance (Qwen size) or from 22.5 % to 47.5 % (InternVL color-position). Fragmentation effects are thus smaller than, and mostly of opposite sign to, what the fragmentation hypothesis predicts.

Z_TV usage suggests model comparisons.

We apologize for the confusion! Indeed, our current wording implies that Z_TV can be used for more comparisons than it actually can be. Given the concerns raised by other reviewers and the fact that we barely utilized the metric in our study, we have decided to fully remove the metric and former Figure 4 from the paper.

Please also specify how the 95% confidence intervals in Figures 5–8 are computed, including the aggregation and resampling unit.

Each error bar in Figs. 5-8 is the unweighted mean accuracy over (feature, model, position) cells, and the error bar is a 95% bootstrap CI (1,000 resamples) over those cells. Stimulus-level paired CIs, which are considerably tighter, are given in the newly added Appendix C for fragmentation. We have added an explanatory sentence to the paper. Thank you for bringing this to our attention!

Please clarify the actual encoder-patch and post-merger visual-token counts.

Thank you for your suggestion! We have added these results to Tab. 7 in the appendix.

For the prompt controls, can the authors provide a comparison under matched batching/inference settings?

Thank you for another great suggestion! We have now added a batched Qwen baseline to Fig. 11/12. The accuracy stays fairly similar: color-position is the only outlier at -5.3%; the single-sample vs. batched patterns share a high Pearson r of 0.94 for that feature. We therefore expect the observed gap to be mostly attributed to the prompt, with a smaller impact of batching. However, globally all prompt patterns still correlate strongly (0.95 and 0.86), and even the lowest feature-specific correlation is moderate (color-position at 0.57). We updated the discussion and added a correlation table (Tab. 3).

Once again, we thank you for your valuable feedback! We look forward to further discussing this if there are remaining questions, and thank you for your careful consideration of our paper.

The paired comparisons show that recognition can vary by position even when the target fits within one patch. Removing the cross-model ranking addresses my concern about the statistic. I’m keeping my rating at 3 because the fragmentation tests cover only three positions.

Dear Reviewer Bg1o,

Thank you for your response. We would like to raise your awareness of the new Appendix C, which already contains a full grid result for fragmentation on Qwen using batching. The results hold, and we are currently running full sweeps without batching for all models. The runs are expected to complete in 90hrs, and we will be happy to add the results to the paper.

Once again, we thank you for your valuable feedback and for your careful consideration of our paper.

Comments on the paper

1 comment

We thank all reviewers for their valuable feedback. We are pleased to hear that all reviewers enjoyed our paper, stating “useful diagnostic/meaningful observations/well motivated” (Bg1o), “I like this result/well motivated” (TgiE), and “interesting/clear and easy to follow” (zCry). Based on the feedback, we have implemented some changes in the PDF to solidify our study (highlighted in yellow), which we discuss in the individual rebuttals below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.