acceptodds
Under review as a conference paper at ICLR 2027

Correct After Rearrangement: A Tile-Permutation Test of Spatial Reliance in Vision-Language Benchmarks

Abstract

Vision-language models are scored on benchmarks such as MMStar, RealWorldQA, and ChartQA, where a score above a text-only baseline is often read as evidence that the model used the spatial arrangement of the image. Removing the image shows which scores need an image, not which properties of that image they need. A model could use the spatial arrangement of the scene, or collect evidence from separate regions independently of their layout. We separate these explanations by cutting each image into equal tiles, permuting the tiles at random, and passing the result to the model's own processor, which preserves the pixel content of every tile and disrupts inter-tile adjacency and tile location. We introduce persistence, the probability that a correct answer survives this intervention, and measure it for three open vision-language models on these three benchmarks. At a two-by-two grid, correctness is retained on about 65% to 80% of permutation trials among originally correct items, and on 44% to 79% at a four-by-four grid. This persistence exceeds content-removed baselines by 17 to 55 percentage points on a matched scale in all 27 model-benchmark-grid combinations, including every combination at the finest grid. Scrambling pixels inside the tiles while keeping the layout fixed reduces accuracy to near the text-only baseline, which places within-tile structure ahead of tile arrangement in this ordering. Most originally correct predictions therefore remain recoverable after the original coarse arrangement is disrupted. We release per-item records so that spatial reliance can be examined item by item.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, multimodal evaluation, spatial reasoning, benchmark analysis, tile permutation, visual robustness, spatial reliance, vision-language benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PdzjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is benchmarking the VLM models ability on how they handle the spatial order permutation of images. The authors propose a method to test it by first split the image into patches, then reorder them. They give the reordered images to the VLM models and test whether VLM models can still provide correct answers. The authors use the group theory to illustrate and explain the results, however, the propositions the authors proved in Section 3.2 are quite simple. Involving the group theory makes the presentation quite unclear. Overall I think this paper is under the acceptance bar of ICLR.

Critical strengths

None

Critical weaknesses

Following are the weaknesses of this paper:

  • Unclear presentation: The presentation of this paper is one of the most critical weaknesses. The authors use complex definitions and proofs for simple concepts. For example, line 162-204 use "orbit statistic" from the group theory to illustrate unordered collection of image patches, then use the group theory to prove that VLMs lose performance with reordered images. I believe this paper isn't a machine learning theory paper introducing new insights from a mathematical view, and it's totally unnecessary to involve those math formulations. Also, as a benchmark paper, this paper lacks enough illustrations and visualizations for the method and the main results, which makes it really hard to read.
  • Minor technical contribution: As a benchmark paper, the authors should reveal new issues of existing models, and ablate where the issues come from. A good benchmark paper can also provide insights on how to solve these issues. This paper doesn't provide any new perspective to the potential drawbacks of current models. Reordering image patches decreases the QA performance and the correlation of split size and performance decrease aren't enough to support an ICLR paper. Also there's no ablation of what are the properties making the performance worse, and there's no qualitative analysis of the results.
  • Unsound method: To analyze the spatial reliance problem, the method provided by the author is to reorder the image patches and evaluate on original QA pairs. I'm not convinced that this is an effective way for spatial reliance evaluation. My main concern is image patch permutation may change the answers to the questions. The authors didn't include any discuss about this in the paper.
Questions for authors

See weakness.

5 comments

Thank you for your review. Your comments led us to make the method more visual and to place the answer-change question at the center of the paper.

1. The main finding

We would like to clarify the main finding, which runs opposite to a performance drop.

  • Most correct answers persist: 65–80% of trials at k=2.
  • Persistence exceeds blind and random-image baselines on the same items by +17.13 to +56.92 pp in all 36 model-benchmark-grid combinations.

Benchmark scores cited as evidence of spatial understanding therefore rest largely on evidence that does not need the image's arrangement. The released per-item records identify the items that do need it.

2. Role of the two propositions (§3.2, now condensed)

The propositions describe what tile permutation leaves available to a model.

  • Proposition 1 bounds what any predictor can achieve from the unordered tiles and the question.
  • Proposition 2 shows that this bound cannot increase as the grid becomes finer.

Together they make the grid size a principled measurement scale and not an arbitrary perturbation strength. Reviewer HhrJ described this as the most elegant part of the paper. In the revision, §3 is shorter, with three short definitions, two propositions, and brief arguments. It also states explicitly that the bound applies to uniform permutations, while persistence uses derangements.

3. "Permutation may change the answer" is now a central result (§1, §5.3, Fig. 3)

A changed answer is what the test is designed to detect. The revision separates why an answer changes. Three quadrant mirrors cut along identical lines and differ only in which relations they reverse:

  • On left/right questions, the left-right mirror costs 26.03 pp more than the top-bottom mirror. On all other questions it costs 1.26 pp more.
  • 82–84% of these failures give the opposite side, so the model reads the rearranged relation.

4. Ablations of the factors that change answers

The revision adds, with Holm correction throughout:

FactorResult
Relation reversed vs. kept, same cuts (Fig. 3)−26.03 vs. −1.26 pp
Inside-quadrant shuffle vs. quadrant mirrors (Fig. 4c)similar on natural images; 6.79–17.84 pp worse on charts
Coarse layout at identical cut lines (Fig. 4d)+2.59 to +8.95 pp, all 12 pairs
Text cut by tile edges, OCR (Fig. 4b)no association (Spearman −0.066 to −0.023)
Pixel shuffle inside tiles (Fig. 4a)most damaging in every pair

These join the existing separator, resolution, retention-strata, category, and label-randomization analyses.

5. Illustrations and qualitative analysis

  • New Figure 1 shows every intervention on one scene, including how a mirror reverses a left-of relation, and new Table 1 lists what each intervention changes and keeps.
  • App. Fig. 8 shows real inputs from each benchmark.
  • Fig. 3c shows the qualitative behavior directly: answers move to the opposite side.

All figures are redesigned. We also added Qwen2.5-VL-7B, which shows the strongest left/right sensitivity of the four models.

6. New perspective for benchmark design

The test gives benchmark authors an item-level filter for arrangement, analogous to MMStar's text-only filter. For example, among RealWorldQA's left/right items, the left-right mirror lowers persistence by 26.45 pp relative to the top-bottom mirror. Such items carry the claim about spatial understanding, and they are now identified without new annotation.

We hope these clarifications and additions address your concerns, and we would welcome any further questions.

Thank you for taking the time to reply. We understand that your score stands. Your review describes our result as "reordering image patches decreases the QA performance," and we think our own presentation led to that reading. Our result is the opposite. Most correct answers stay correct after the patches are reordered. We would be grateful if you could look once more at the contribution, which we state below as simply as we can.

The question

When a model answers a benchmark question correctly, did the model need to know where things are in the image?

The test

We cut the image into four pieces, shuffle them like a jigsaw puzzle, and ask the same question. Every pixel remains, and only the positions change.

What is new

The paper contributes four things.

1. A measurement that existing checks do not provide

CheckWhat is taken awayWhat the result shows
No-image baseline (existing)The whole imageWhether a question needs an image
Tile permutation (ours)Only the arrangementWhether a question needs the arrangement

Tile permutation gives, to our knowledge, the first measurement of how much of an official benchmark score depends on the arrangement of an image, as distinct from its content.

2. A finding about what benchmark scores mean

With four shuffled pieces, 65% to 80% of correct answers stay correct. Across all 36 model-benchmark-grid combinations, the share that stays correct is 17 to 57 percentage points above what the same models reach on the same questions with no image or an unrelated image. Most of the score on MMStar, RealWorldQA, and ChartQA is earned without the original arrangement, and a high score on these benchmarks does not by itself show that a model used spatial arrangement.

3. A demonstration that the test reacts where arrangement matters

A left-right mirror and a top-bottom mirror of the four pieces cut the image along the same lines and keep every piece intact. On questions about left and right, the left-right mirror costs 26.03 points more than the top-bottom mirror, against 1.26 points on all other questions. Where the answer names a side, 82% to 84% of the failures give the opposite side, which means the model read the rearranged image and answered accordingly. The concern in your review, that a permutation may change the answer, is the quantity measured here.

4. A tool for benchmark authors

The per-item records identify which benchmark questions depend on arrangement, with no new annotation. Authors can select those questions when the goal is to measure spatial understanding.

The argument in four lines

  1. An answer that needs the arrangement breaks when the arrangement is shuffled.
  2. Most correct answers do not break.
  3. The test is not blind, because left-right answers break when left and right are mirrored, and they break toward the opposite side.
  4. Most correct answers on these benchmarks were therefore recoverable without the original arrangement.

We would be glad to answer any specific question on these results.

  1. I wonder why the result is opposite? I think "decrease" is the right word to describe the results. Is the performance of the reordered image at least on-par as the original images, or even higher?
  2. For the contribution part, "An answer that needs the arrangement breaks when the arrangement is shuffled." is quite obvious, and "Most correct answers do not break." is also obvious when the results don't depend on the spatial relationship. I still don't think these contribution reach the bar of ICLR.

Given the two rounds of discussions with the authors, I'm pretty sure my final decision to this paper is clear rejection.

Reviewer_HhrJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

he paper proposes a tile-permutation test as a model-agnostic diagnostic for how much of a vision–language model's benchmark score depends on the spatial arrangement of an image, as opposed to the mere presence of its contents. For a benchmark item, the image is reflection-padded, cut into a k×k grid (k∈{2,4,8}), its tiles are randomly permuted in pixel space, and the result is passed through the model's released processor; model internals are untouched. The authors define persistence as the fraction of originally-correct items that remain correct after this permutation, and formalize an orbit-based upper bound (Prop. 1) on any permutation-invariant predictor's accuracy together with a nested-grid monotonicity (Prop. 2) that makes k a scale on which spatial reliance can be measured. On three open VLMs (LFM2.5-VL-1.6B, SmolVLM2-2.2B, Qwen2.5-VL-3B) and three benchmarks (MMStar, RealWorldQA, ChartQA), persistence at k=2 is 65–80%; the target-specific persistence gap over content-removed baselines (blind text-only, random-image substitution) is +17 to +55 pp in all 27 model×benchmark×grid cells, and all 95% intervals exclude zero. Within-tile pixel scrambling, in contrast, drops accuracy to near the blind baseline, placing within-tile structure ahead of tile arrangement in these benchmarks. Controls include a one-pixel mean-color separator ablation, a 4× resolution control for Qwen at k=8, a label-randomization Monte Carlo null (p=0.0002 for all 27 contrasts, Bonferroni-adjusted), and a retention-stratified analysis across tile-adjacency survival (zero/low/high). Per-item records, permutation seeds, and correctness labels are released.

Critical strengths
  1. The nested-grid orbit bound (Prop. 1 + Prop. 2) turns k into a measurement scale, not a tuning knob. This is the most elegant piece of the paper. Showing that BayesAcc_D(O_k₂, Q) ≤ BayesAcc_D(O_k₁, Q) for nested k₁ < k₂ — because every k₁-tile deterministically decomposes into its k₂-tiles — gives the k-sweep a principled interpretation as a monotone reduction of positional information, not a sweep over an ad-hoc hyperparameter. Combined with the fact that the intervention is applied before any model-specific processor and leaves internals untouched, this gives the diagnostic a cleaner identification story than prior order-perturbation work (e.g., Directed-Tokens, patch-dropout) that touches internal visual tokens and therefore couples to tokenizer/architecture choices.
  2. Scale-matched content-removed baselines with target-specific gap. Measuring persistence against blind and random-image baselines restricted to the same originally-correct subset, and computing a per-item target-specific gap (TSP), places models with very different native accuracies on one axis and controls for the obvious "well, maybe random guessing would also work" alternative. The +17 to +55 pp TSP with all 95% intervals excluding zero in all 27 cells is a tight, falsifiable headline that most prior "does the image matter?" studies cannot deliver because they contrast with an entirely different image rather than a scale-matched permutation of the same image.
Critical weaknesses
  1. Model scope undercuts the paper's framing. The paper's claims are stated for "vision-language models" and "vision-language benchmarks," but the empirical base is three open models at 1.6B–3B parameters. The benchmarks (MMStar, RealWorldQA, ChartQA) are routinely cited for frontier closed models (GPT-4V/4o, Claude, Gemini) and large open models (Qwen2.5-VL-72B, InternVL-78B), none of which are evaluated here. Persistence patterns at this scale may not transfer, and the paper does not provide intermediate evidence (e.g., a 7B/13B anchor from the same family as Qwen2.5-VL-3B) to argue they do. For a diagnostic paper whose contribution is the methodology plus the empirical result, the empirical half's reach is limited.
  2. Within-tile shuffle is interpreted beyond what the paper itself admits is warranted. §5.3 reads: "Within-tile pixel scrambling therefore produces a substantially larger reduction in accuracy than whole-tile permutation... which places within-tile structure ahead of tile arrangement in the ordering these measurements support." The Limitations section then says: "the relative size of the two drops should not be read as a quantitative decomposition of local-content and spatial-layout contributions; interventions of matched severity would be needed for that." These two statements point in different directions, and §5.3's ordering is one of the more consequential interpretive moves in the paper. Either (a) a matched-severity local perturbation (e.g., tile-local permutation of 2×2 sub-tiles within each k=2 tile, or local Gaussian texture scrambling calibrated to a matched information-destruction rate) should be added, or (b) the §5.3 ordering should be softened to match the Limitations.
Questions for authors
  1. Can the within-tile vs. tile-permutation comparison in §5.3 be matched for perturbation severity, for example by a local permutation of 2×2 sub-tiles within each k=2 tile (which destroys the same amount of pixel-level structure as a k=4 tile permutation but keeps the coarse 2×2 layout intact)? Without such a matched intervention, the "within-tile structure ahead of tile arrangement" ordering is confounded with perturbation strength, as your own Limitations section notes.
  2. At least one measurement on a 7B-scale model (e.g., Qwen2.5-VL-7B, which is the same family as your 3B anchor and has the same processor) would substantially raise confidence in the pattern. Was this considered? If the TSP collapses at 7B, the paper's framing changes materially.
  3. The separator-line ablation controls for one artifact of the intervention (added 1-pixel edges at tile boundaries) but not for text/glyph fragmentation when tile cuts pass through axis labels or numeric values. On ChartQA at k=8, approximately what fraction of tile boundaries pass through a text region in the ground-truth OCR? And is the k=8 ChartQA decline (33–40%) partially driven by broken glyphs rather than disrupted mark arrangement?
  4. Figure 3 reports MMStar persistence per capability category. Was a family-wise correction applied to the 18 category means or the 15 within-model category-vs-category contrasts? With ~250 items per category, the ordering (coarse perception high, fine-grained perception low) is plausible, but the magnitudes of the differences would benefit from a reported adjustment.
  5. Prop. 1 bounds the expected accuracy of any predictor measurable with respect to (O_k(X), Q) under uniform permutations. The reported quantity, however, is persistence under derangements (fixed-point-free). Prop. 1 applies to the uniform group S_{k²} that includes fixed points. Could you clarify to what extent the bound still orders persistence (as opposed to raw accuracy) across k, and whether derangement-restricted persistence satisfies an analogous nested-grid monotonicity?
  6. Table 6 (target-specific persistence gap, the main quantitative result cited across §5–§6) is in the appendix, while Table 1 in the main text shows persistence alongside blind/random/within-tile accuracies on a different scale. Consider moving Table 6 (or an abridged version) into the main text so the headline +17 to +55 pp claim is directly auditable where it is first stated.
2 comments

Thank you for the thorough and constructive review, and for highlighting the nested-grid bound and the scale-matched gap. We ran both experiments you proposed and answered the remaining questions. Each change is in the revised PDF.

Q1. Matched-severity control (§4.1, §5.4, Fig. 4, App. A.16)

We implemented the control you suggested: a derangement of the 2×2 sub-tiles inside each quadrant, which keeps the coarse 2×2 layout intact. The paper calls it the inside-quadrant shuffle (Table 1), and we crossed it with the quadrant mirrors on the same 4×4 grid. The two factors are complementary in severity. The mirrors keep 77.8% of directed adjacencies and displace tiles by 2.67 cells. The inside-quadrant shuffle keeps 18.3% and displaces tiles by 1.33 cells.

Contrast (12 model-benchmark pairs)Result
Inside-quadrant shuffle minus mirrors, MMStar/RWQA−2.37 to +5.47 pp (1 of 8 significant after Holm, where the shuffle costs less)
Same, ChartQA−6.79 to −17.84 pp (all 4 significant)
Same 8×8 cut lines, inside-quadrant shuffle minus full derangement+2.59 to +8.95 pp, all 12 significant
Pixel shuffle inside quadrantslowest everywhere (16.49–55.30%)

This decomposition now replaces the ordering sentence of the original §5.3 (now §5.4), so the Limitations note on matched severity is no longer needed. Local structure and coarse placement cost comparable amounts on natural images. Local structure costs more on charts. Coarse placement contributes at identical cuts in every model and benchmark.

Q2. A 7B model

We added Qwen2.5-VL-7B (same family and processor) under every condition of the study. The target-specific persistence gap remains large.

  • TSP ranges from +20.80 to +56.92 pp, and all intervals exclude zero. All 36 combinations are positive.
  • k=2 persistence is 71.68–73.92%, within the range of the smaller models, while 7B has the highest native accuracy on every benchmark.
  • 7B shows the strongest left/right sensitivity in the new mirror test. Its RealWorldQA contrast is −35.76 pp, and 58% of its left/right trials on MMStar and RealWorldQA give the opposite side (§5.3).

Q3. Glyph fragmentation on ChartQA (§5.4, Fig. 4b, App. A.17)

We ran EasyOCR on all 1,250 images:

GridBoundary segments through textText boxes cutItems with cut text
k=243.36%14.41%85.12%
k=428.78%34.24%98.40%
k=822.51%63.21%100%

k=8 persistence is flat in the share of boundaries crossing text (Spearman −0.066 to −0.023 across models). The same-cut contrast (inside-quadrant shuffle minus full derangement at 8×8) has a positive point estimate in every tercile of every model. The k=8 decline therefore reflects lost arrangement, beyond broken glyphs.

Q4. Family-wise correction for categories

We applied Holm correction over the 15 within-model contrasts. The coarse-versus-fine-grained perception difference (13.41–18.22 pp) is significant in all four models, with adjusted p from 0.003 to 0.016.

Q5. Proposition 1 and derangements

Proposition 1 is stated for uniform permutations, and we report exactly that quantity (Fig. 2, top row; App. Table 4). It decreases monotonically from k=2 to k=8 in all 12 model-benchmark pairs, the direction Proposition 2 gives for the bound. Persistence uses derangements so that every tile moves, and it decreases monotonically in all 12 pairs as well. Because derangements do not form a group, and a deranged image also reveals that no tile sits in its original position, the revised §3.2 reports the two quantities side by side and states how they relate.

Q6. Table 6 in the main text

We moved it. Table 2 now reports TSP for all 36 combinations next to persistence. App. Table 6 gives the intervals.

We believe the two experiments you proposed now address both points you raised, and we are happy to provide any further detail.

Since the discussion period is closing soon and we may not be able to add further analyses, we would like to leave a short final note. We are very grateful for your careful and constructive review. Regardless of the final outcome, your suggestions on matched severity and model scale improved the paper considerably and have pointed us to our next research question. We sincerely appreciate the time and effort you put into it.

Reviewer_CGPsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper claims that it evaluates how much VLMs rely on spatial information by dividing images into tiles, randomly rearranging them, and measuring how often models can still answer the original questions correctly. Experiments on three VLMs across MMStar, RealWorldQA, and ChartQA show that a substantial proportion of correct predictions persist despite disrupted spatial arrangements.

Critical strengths

The proposed tile-permutation method is easy to implement, without additional annotations or model modifications, and can be applied across different VLMs and benchmarks. But how to use it to deliver good insight is a different story.

Critical weaknesses

Insufficient motivation and questionable validity of the proposed evaluation.

I am not convinced that the proposed tile-permutation intervention provides a meaningful measure of spatial reliance. The authors argue that the intervention preserves visual content because all original pixels are retained. However, preserving pixels does not necessarily preserve semantic information or the validity of the original question-answer pair. For exmaple, if the question is "Is the dog to the left of the cat?", after rearrangement, the relationship may change, or the objects may be fragmented, the question is no longer answerable. Or like question in ChartQA, rearranging the tiles can totally mess up the labels, bars and axes, making the question not answerable or the original correct answer is wrong. It is unclear whether the proposed metric captures spatial reliance or merely robustness to an artificial image corruption.

The presentation and writing of this paper is far below the standard of ICLR.

Questions for authors

See above.

3 comments

Thank you for the careful review. We also appreciate that you recognized the test needs no annotation or model modification and applies across VLMs and benchmarks. Your central question is whether persistence captures spatial reliance or only robustness to an artificial corruption. The revision answers it with a new experiment.

1. Your dog/cat example is what the test is built to detect, and the revision attributes the change directly

If the answer depends on the left-right relation, rearranging the image should change it. What matters is separating that effect from the damage done by the cut. The revised paper adds a quadrant-mirror test (§5.3, Fig. 3). Three quadrant mirrors (left-right, top-bottom, and both, which is a 180° rotation) cut the image along the same lines and keep every quadrant intact. They differ only in which relations they reverse. If persistence measured robustness to corruption, the two mirrors would cost the same. They do not:

Pooled over 4 models and 3 benchmarksL-R mirror minus T-B mirror
Left/right questions (295 items)−26.03 pp [−31.28, −20.88]
All other questions (2,487 items)−1.26 pp [−2.63, +0.03]
Difference−24.77 pp, p = 0.0002

The failures also have a direction. Among left/right questions whose answer names a side, 82–84% of the failures under the L-R mirror give the opposite side. The opposite side appears in 32–58% of trials, against 5–12% under the T-B mirror. The model reads the rearranged relation and answers accordingly. This is spatial reliance detected item by item, with the corruption held fixed.

2. ChartQA labels, bars, and axes

We measured directly how often the cuts pass through text (§5.4, Fig. 4b, App. A.17), running OCR on all 1,250 ChartQA images.

  • At k=8, 63.21% of text boxes are cut. Yet k=8 persistence is flat between items with little and much cut text (Spearman −0.066 to −0.023 across the four models).
  • A same-cut comparison isolates layout. With identical 8×8 cut lines, keeping sub-tiles inside their quadrants raises persistence by +2.59 to +8.95 pp in all 12 model-benchmark pairs (all significant after Holm correction). The loss under full permutation is therefore lost arrangement, beyond fragmentation.
  • At k=2, where cuts are coarse, ChartQA persistence is 65–72%. The gap over content-removed baselines on the same items is +44.83 to +56.92 pp.

3. Which answers depend on the arrangement?

An answer should change only where it depends on the rearranged relation, and the identical-cut comparisons above locate those items. The paper's headline concerns the complementary case. When a correct answer persists after the layout is scrambled, that answer was recoverable without the original arrangement. This holds for 65–80% of trials at k=2, and persistence exceeds blind and random-image baselines on the same items by +17.13 to +56.92 pp in all 36 model-benchmark-grid combinations. The two results together form a two-sided diagnostic: the test identifies which correct answers needed the arrangement and which did not.

4. Presentation

To make the argument easier to follow, the revision makes these changes:

  • A new Figure 1 illustrates every intervention.
  • A new Table 1 summarizes what each intervention changes and keeps.
  • The target-specific gap now sits in Table 2.
  • §3 is condensed.
  • The main figures are redesigned.
  • A fourth model, Qwen2.5-VL-7B, is added.

We would be glad to clarify anything further.

Thank you for the rebuttal. I still believe the paper's presentation, practical insights, and implications for benchmark design need SIGNIFICANT development and validation. The current version is far below the ICLR standard. I will keep my score of 1.

Since the discussion period is coming to an end and we may not be able to provide further analyses, we would like to leave a brief final note. Thank you for the time and effort you put into reviewing our paper. Whatever the final decision, your review and this discussion have been very helpful to us, and your question about whether the test separates spatial reliance from damage to the image has shaped the direction of our next study. We truly appreciate your thoughtful review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their time and constructive comments. The revision adds four experiments and restructures the presentation. All models and conditions use the same items, code, and permutation seeds. Re-evaluating the three original models reproduces the submitted numbers within 0.92 pp (padded accuracy), 0.40 pp (k=8 persistence), and 0.35 pp (pixel-shuffle accuracy).

New experiments

Question raisedNew experimentResult
Does persistence measure spatial reliance or robustness to corruption? (CGPs, Pdzj)Quadrant mirrors (§5.3, Fig. 3). Left-right and top-bottom mirrors of the quadrants make identical cuts and reverse different relationsOn left/right questions the L-R mirror costs 26.03 pp more than the T-B mirror; on all other questions 1.26 pp. 82–84% of the resulting failures name the opposite side
Matched-severity control for local structure vs. arrangement (HhrJ)Inside-quadrant shuffle in a factorial design with the mirrors, plus a same-cut comparison (§5.4, Fig. 4)The inside-quadrant shuffle costs about as much as mirroring quadrants on natural images and 6.79–17.84 pp more on charts. At identical 8×8 cuts, keeping quadrants in place adds +2.59 to +8.95 pp in all 12 pairs
Glyph fragmentation on ChartQA (HhrJ, CGPs)OCR of cut text (§5.4, Fig. 4b)63.21% of text boxes are cut at k=8, yet k=8 persistence is flat in the share of cut text (Spearman −0.066 to −0.023)
Model scale (HhrJ)Qwen2.5-VL-7B under every condition of the studyTSP +20.80 to +56.92 pp, all intervals exclude zero; strongest left/right sensitivity of the four models

Presentation

  • New Figure 1 illustrates every intervention on one scene.
  • A new Table 1 summarizes what each intervention changes and keeps.
  • The target-specific persistence gap is now in Table 2 for all 36 model-benchmark-grid combinations.
  • §3 is condensed and states explicitly which quantity Proposition 1 bounds.
  • All figures are redesigned, and family-wise (Holm) corrections are reported for every family of tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.