- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is benchmarking the VLM models ability on how they handle the spatial order permutation of images. The authors propose a method to test it by first split the image into patches, then reorder them. They give the reordered images to the VLM models and test whether VLM models can still provide correct answers. The authors use the group theory to illustrate and explain the results, however, the propositions the authors proved in Section 3.2 are quite simple. Involving the group theory makes the presentation quite unclear. Overall I think this paper is under the acceptance bar of ICLR.
Critical strengths
None
Critical weaknesses
Following are the weaknesses of this paper:
- Unclear presentation: The presentation of this paper is one of the most critical weaknesses. The authors use complex definitions and proofs for simple concepts. For example, line 162-204 use "orbit statistic" from the group theory to illustrate unordered collection of image patches, then use the group theory to prove that VLMs lose performance with reordered images. I believe this paper isn't a machine learning theory paper introducing new insights from a mathematical view, and it's totally unnecessary to involve those math formulations. Also, as a benchmark paper, this paper lacks enough illustrations and visualizations for the method and the main results, which makes it really hard to read.
- Minor technical contribution: As a benchmark paper, the authors should reveal new issues of existing models, and ablate where the issues come from. A good benchmark paper can also provide insights on how to solve these issues. This paper doesn't provide any new perspective to the potential drawbacks of current models. Reordering image patches decreases the QA performance and the correlation of split size and performance decrease aren't enough to support an ICLR paper. Also there's no ablation of what are the properties making the performance worse, and there's no qualitative analysis of the results.
- Unsound method: To analyze the spatial reliance problem, the method provided by the author is to reorder the image patches and evaluate on original QA pairs. I'm not convinced that this is an effective way for spatial reliance evaluation. My main concern is image patch permutation may change the answers to the questions. The authors didn't include any discuss about this in the paper.
Questions for authors
See weakness.
5 comments
Thank you for taking the time to reply. We understand that your score stands. Your review describes our result as "reordering image patches decreases the QA performance," and we think our own presentation led to that reading. Our result is the opposite. Most correct answers stay correct after the patches are reordered. We would be grateful if you could look once more at the contribution, which we state below as simply as we can.
The question
When a model answers a benchmark question correctly, did the model need to know where things are in the image?
The test
We cut the image into four pieces, shuffle them like a jigsaw puzzle, and ask the same question. Every pixel remains, and only the positions change.
What is new
The paper contributes four things.
1. A measurement that existing checks do not provide
| Check | What is taken away | What the result shows |
|---|---|---|
| No-image baseline (existing) | The whole image | Whether a question needs an image |
| Tile permutation (ours) | Only the arrangement | Whether a question needs the arrangement |
Tile permutation gives, to our knowledge, the first measurement of how much of an official benchmark score depends on the arrangement of an image, as distinct from its content.
2. A finding about what benchmark scores mean
With four shuffled pieces, 65% to 80% of correct answers stay correct. Across all 36 model-benchmark-grid combinations, the share that stays correct is 17 to 57 percentage points above what the same models reach on the same questions with no image or an unrelated image. Most of the score on MMStar, RealWorldQA, and ChartQA is earned without the original arrangement, and a high score on these benchmarks does not by itself show that a model used spatial arrangement.
3. A demonstration that the test reacts where arrangement matters
A left-right mirror and a top-bottom mirror of the four pieces cut the image along the same lines and keep every piece intact. On questions about left and right, the left-right mirror costs 26.03 points more than the top-bottom mirror, against 1.26 points on all other questions. Where the answer names a side, 82% to 84% of the failures give the opposite side, which means the model read the rearranged image and answered accordingly. The concern in your review, that a permutation may change the answer, is the quantity measured here.
4. A tool for benchmark authors
The per-item records identify which benchmark questions depend on arrangement, with no new annotation. Authors can select those questions when the goal is to measure spatial understanding.
The argument in four lines
- An answer that needs the arrangement breaks when the arrangement is shuffled.
- Most correct answers do not break.
- The test is not blind, because left-right answers break when left and right are mirrored, and they break toward the opposite side.
- Most correct answers on these benchmarks were therefore recoverable without the original arrangement.
We would be glad to answer any specific question on these results.
- I wonder why the result is opposite? I think "decrease" is the right word to describe the results. Is the performance of the reordered image at least on-par as the original images, or even higher?
- For the contribution part, "An answer that needs the arrangement breaks when the arrangement is shuffled." is quite obvious, and "Most correct answers do not break." is also obvious when the results don't depend on the spatial relationship. I still don't think these contribution reach the bar of ICLR.
Given the two rounds of discussions with the authors, I'm pretty sure my final decision to this paper is clear rejection.
Thank you for your review. Your comments led us to make the method more visual and to place the answer-change question at the center of the paper.
1. The main finding
We would like to clarify the main finding, which runs opposite to a performance drop.
Benchmark scores cited as evidence of spatial understanding therefore rest largely on evidence that does not need the image's arrangement. The released per-item records identify the items that do need it.
2. Role of the two propositions (§3.2, now condensed)
The propositions describe what tile permutation leaves available to a model.
Together they make the grid size a principled measurement scale and not an arbitrary perturbation strength. Reviewer HhrJ described this as the most elegant part of the paper. In the revision, §3 is shorter, with three short definitions, two propositions, and brief arguments. It also states explicitly that the bound applies to uniform permutations, while persistence uses derangements.
3. "Permutation may change the answer" is now a central result (§1, §5.3, Fig. 3)
A changed answer is what the test is designed to detect. The revision separates why an answer changes. Three quadrant mirrors cut along identical lines and differ only in which relations they reverse:
4. Ablations of the factors that change answers
The revision adds, with Holm correction throughout:
These join the existing separator, resolution, retention-strata, category, and label-randomization analyses.
5. Illustrations and qualitative analysis
All figures are redesigned. We also added Qwen2.5-VL-7B, which shows the strongest left/right sensitivity of the four models.
6. New perspective for benchmark design
The test gives benchmark authors an item-level filter for arrangement, analogous to MMStar's text-only filter. For example, among RealWorldQA's left/right items, the left-right mirror lowers persistence by 26.45 pp relative to the top-bottom mirror. Such items carry the claim about spatial understanding, and they are now identified without new annotation.
We hope these clarifications and additions address your concerns, and we would welcome any further questions.