acceptodds
Under review as a conference paper at ICLR 2027

Hidden Sensitivity in Spatial Reasoning Benchmarks: Diagnosis and Factor-Balanced Evaluation

Abstract

Spatial reasoning benchmarks for vision-language models (VLMs) report aggregate scores that implicitly weight evaluation conditions by their frequency in the benchmark. Across four video spatial reasoning benchmarks (VSI-Bench, ReVSI, VSTI-Bench, and OSI-Bench) and 14 VLMs, we show that the distributions of task-agnostic factors, such as ground-truth answer value and queried object type, are highly skewed. Models exhibit distinct performance patterns across these factors, and plausible changes in composition alone often reorder leaderboards. We quantify this effect through a model’s composition advantage, defined as the difference between its observed score and a factor-balanced counterpart. To identify consequential factors, we introduce the Composition Sensitivity Test (CST), an information-theoretic diagnostic that retains factor-performance dependencies that persist after adjustment for entangled factors. It identifies answer value and object type as recurring sources of composition sensitivity, while also capturing additional influential factors when present. The Factor-Balanced Score (FBS) then reweights benchmark items to alleviate the dominance of frequently represented levels of diagnosed factors in the aggregate score. Across the four benchmarks, balancing changes the ranks of 53% of model-task pairs and the top-ranked model on 10 of 29 affected tasks. On object counting, for example, answers are concentrated in a narrow range of values; on VSI-Bench, balancing this distribution decreases a spatially fine-tuned model’s score by 0.25, while a general-purpose model whose score barely changes rises from eighth to first place. This illustrates how benchmark compositions can unintentionally favor some models over others. Moreover, under systematic composition changes, FBS reduces score variability by 63-71% relative to the observed scores. Together, these results show that benchmark composition can notably affect conclusions about spatial reasoning performance of VLMs. We therefore advocate jointly reporting observed scores, balanced scores, and composition advantages to make this influence explicit.

Primary area:
datasets and benchmarks
Keywords:
spatial reasoning, vision-language models, multimodal evaluation, benchmark robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1iqoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the composition of spatial reasoning benchmarks—distributions of task-agnostic factors such as ground-truth answer value, queried object type, room type, or correct option—shapes aggregate scores and leaderboard rankings. The aggregate score is decomposed as a frequency-weighted average over factor configurations (Eq. 2). Two tools are introduced: (1) the Composition Sensitivity Test (CST), which uses marginal vs. adjusted (conditional) mutual information between discretized per-item scores and candidate factors, combined with a magnitude threshold and a ratio band, to classify factors as direct contributors / suppressed / surface correlates / negligible; and (2) the Factor-Balanced Score (FBS), which reweights items via stable balancing weights so that diagnosed direct contributors have approximately uniform marginals, yielding a balanced score and a per-model "composition advantage" (observed − balanced). Across 4 video spatial benchmarks, 39 tasks, and 14 VLMs, CST flags answer value (26/39 tasks) and object type (17/39) as the dominant contributors; balancing changes 53.4% of model-task ranks and the top-1 model on 10/29 balanced tasks; spatially fine-tuned models often fall below their own backbones after balancing; and FBS reduces score variability by 63–71% under systematic composition resampling.

Critical strengths
  1. An important and understudied validity threat, crisply formalized. The observation that benchmark scores are composition-weighted averages—and that skewed answer-value/object-type distributions can decide leaderboard winners—is made concrete with a clean decomposition (Eq. 2) and striking worked examples (e.g., VSI-Bench counting, where balancing moves Qwen3-VL-8B from eighth to first and shrinks SpatialLadder's lead over Qwen2.5-VL-3B from 0.38 to 0.03).
  2. Methodologically principled. Using CMI to adjust for factor entanglement addresses the obvious objection that factor–score associations are confounded; using stable balancing weights (Zubizarreta, 2015) rather than naive inverse-frequency weighting controls weight dispersion. The association-not-causation caveat for "direct contributor" is stated honestly.
  3. Scale and consistency of the empirical evidence. Four benchmarks, 39 tasks, 14 models, with per-regime diagnosis (Fig. 3), full re-ranking statistics, composition-advantage heatmaps, and a systematic resampling protocol. The finding that relational tasks have no direct contributor while counting/quantitative tasks are dominated by answer value is a useful, actionable map for benchmark designers.
  4. Actionable recommendation. Reporting (observed, balanced, composition advantage) jointly is cheap to adopt and immediately interpretable; the CA metric nicely exposes who benefits from a benchmark's quirks (e.g., spatially fine-tuned models gaining +0.16 to +0.25 on VSI-Bench counting purely from composition).
Critical weaknesses
  1. The robustness evaluation is partly circular. The composition-resampling experiment perturbs compositions along the top-1 direct contributor and shows FBS scores/rankings vary less. But FBS is constructed to flatten exactly that factor's marginal influence, so reduced sensitivity to resampling along it is close to guaranteed by design. A more convincing robustness test would perturb along factors not balanced (e.g., the second-ranked contributor, or held-out factors like brightness), or subsample items uniformly at random (which changes all factor marginals jointly) and check whether FBS-on-subset A agrees with FBS-on-subset B more than observed scores do.
  2. The uniform-marginal target is doing normative work that the paper under-interrogates. FBS answers "what would the ranking be if factor levels were equally represented?"—not "what is the true ranking." The headline re-ranking results (53.4% of ranks, 10/29 winners) are rhetorically framed as exposing misleading leaderboards, but an equally valid reading is that the balanced ranking is simply a different ranking whose target distribution is no more grounded in deployment reality. The limitation section acknowledges this, but the abstract/main text ("unintentionally favor", "alleviate the dominance") leans toward treating balanced as better. For counting tasks in particular, low-count dominance may reflect the real frequency of objects in scenes, not a construction artifact.
  3. MI/CMI estimation on small, discretized samples is fragile and not stress-tested. Per-task item counts range from ~100–900; Y is discretized into 10 equal-width bins; factors like object type have many levels. Empirical MI is notoriously biased upward at these sample sizes, and CMI even more so. The sensitivity analysis covers the threshold constant c (App. E), but I could not find permutation-based null distributions or bias-corrected MI estimators—without them, the direct/suppressed/negligible classifications may inherit estimation noise.
  4. Marginal balancing ignores joint configurations. FBS equalizes each diagnosed factor's marginal, but entangled factors mean rare joint cells (e.g., a specific room type × answer value) can receive extreme weights. SBW limits dispersion, but no effective sample size or weight-distribution statistics are reported per task, so it is hard to judge how much of each benchmark effectively survives reweighting.
  5. Scope is narrow relative to the framing. All four benchmarks are recent video spatial-reasoning suites with similar construction philosophies. Claims about "spatial reasoning benchmarks" generally—and the recommendation's portability to e.g. image-based or embodied benchmarks—would benefit from at least one out-of-family demonstration.
Questions for authors

See weaknesses above

4 comments

We sincerely thank you for the thorough and constructive review, and for recognizing that we address “an important and understudied validity threat, crisply formalized” with a “methodologically principled” approach.


Revision with updated factor definitions. Based on the thoughtful suggestions from Reviewers wt8n and 7J5Z on more reliable factor definitions, we made the following two changes, re-ran the full CST → FBS pipeline on all 39 tasks, and revised the paper accordingly:

  • “Difficulty proxy” based on Kimi-K3. The former “object type” was more of a per-item difficulty measure than an object category. We now compute it with Kimi-K3, which is not among the evaluated models, and rename it “difficulty proxy”.
  • A new “object category” factor. We added a new factor, “object category”, derived by extracting the objects in the text query of each question and grouping them into semantic categories.

This makes our analysis fairer and more reliable, as it removes a possible bias from computing the difficulty proxy (the former “object type”) with Qwen2.5-VL-7B, one of the evaluated models. With the revised factors, the main conclusions still hold: balancing changes the top-ranked model on 8 of 29 tasks (previously 10) and 41.4% of model-task ranks (previously 53.4%). All revised text, tables, and figures are highlighted in color in the revised PDF, and we would be grateful if you could take a look at these parts. The two factors are defined in Section 3.1 and Appendix D.


[W1] Robustness evaluation

Thank you for this suggestion. We ran both suggested tests and one additional test (Table 3-1); their protocols and further variants are in the new Appendix I.

Table 3-1. Ranking agreement between two subsets (mean Kendall's τ of the 14-model rankings).

TestSubsets differ inτ: observedτ: FBS
Shift along the top balanced factorcomposition0.660.83
Shift along the top unbalanced factorcomposition0.820.83
Random disjoint halvessampled items only0.760.69
Shift along a random combination of all factorscomposition0.740.80

Random halves measure sampling noise, not composition. A random split keeps the distribution of every factor the same in both halves in expectation (mean change of a factor's marginal: 0.07, vs. 0.22 along the balanced factor in Section 4.3). The two rankings then differ only in which items are drawn, and FBS pays the variance cost of reweighting (0.76 → 0.69). The problem we address is the opposite case: subsets whose factor distributions differ.

We therefore also shift composition along random directions. Instead of the top-1 factor, each composition is tilted along a random linear combination of all top-10 candidate factors, redrawn every time, so all factor distributions change jointly, in directions CST does not choose. FBS raises agreement from 0.74 to 0.80 (higher on 24 of 29 tasks) and halves the score SD (0.045 → 0.023). The gain comes from the diagnosed factors: along unbalanced factors alone, FBS neither helps nor hurts (0.82 → 0.83 above; 0.79 → 0.80 in Appendix I).


[W2] Uniform-marginal target

(1) “An equally valid reading is that the balanced ranking is simply a different ranking whose target distribution is no more grounded in deployment reality.”

We agree, and first refer the reviewer to Appendix A (“Does FBS provide universally more faithful model comparisons?” and “Why use a uniform reference for FBS?”). FBS answers “what would the ranking be if factor levels were equally represented?”, not “what is the true ranking”, and we do not claim that equal weight is the “correct” target. Our motivation is the composition bias introduced when benchmark data are acquired.

Choosing the right target remains an open question. Spatial quantities vary widely across environments (Table A.1: mean object size 13–390 cm, mean object count about 1–2 to 56), so any single dataset distribution would privilege one environment. For example, 75% of ReVSI object sizes lie between 25 and 150 cm, while the mean object size is 13 cm in YCB and 390 cm in KITTI (Table A.1). Without a justified external target, we use the uniform target as a transparent, frequency-neutral reference within the benchmark’s support; when a task-specific target is available, our framework can balance toward it.

(2) “For counting tasks in particular, low-count dominance may reflect the real frequency of objects in scenes, not a construction artifact.”

We tested real-frequency targets directly. We used the mean values of the real data corpora in Table A.1 as answer-value targets, rescaling each task’s answer distribution to the corpus mean with its shape and coefficient of variation fixed (revised factors; see our opening comment). First, the four counting tasks are less concentrated on low counts than real indoor scenes (mean answer 2.6–7.0 vs. 1.3–2.1), so their composition is not a direct record of real object frequencies. Second, re-ranking under real-frequency targets is even larger than under the uniform target (84–89% vs. 73% of model ranks; Table 3-2). Including object size, 72% of model ranks and the top-1 model in 15 of 19 task–target combinations change, so the reversals are not specific to the uniform target.

Table 3-2. Object counting rankings (4 tasks, 14 models) under answer-value targets from Table A.1, compared with the observed ranking.

TargetMeanModel ranks changedTop-1 changed
Uniform (FBS)–41 / 56 (73%)2 / 4
Indoor scan datasets (low end)1.347 / 56 (84%)3 / 4
Indoor scan datasets (high end)2.150 / 56 (89%)4 / 4
FSC-1475647 / 56 (84%)2 / 4

(3) “The abstract/main text […] leans toward treating balanced as better.”

We have revised the wording accordingly, presenting the changed ranks (41.4%) and winners (8/29) as the sensitivity of comparisons to composition, not as leaderboard errors:

  • “alleviate the dominance of frequently represented levels of diagnosed factors” → “make the observed levels of diagnosed factors contribute equally”.
  • “can unintentionally favor some models over others” → “can decide which model ranks first”.

[W3] Reliability of the MI/CMI estimates

Bias correction and permutation nulls are part of CST. MI and CMI are estimated with the Miller-Madow correction (within each stratum for CMI), and c is calibrated against a permutation null (200 score shuffles per task): at c = 0.02, the resulting threshold γₜ = cH(Yₜ) exceeds the 95th percentile of the chance MI of every direct contributor on all tasks with N ≥ 200. We have made this explicit in Section 3.2. We also tested each of the 69 direct factor–task pairs against a per-pair permutation null for MI and a within-strata null for CMI: 65 pass both (median ratio to the 95th percentile: 7.3 for MI, 5.6 for CMI).

Discretization. With 5 or 3 score bins instead of 10, the selected factor sets change on about half of the tasks, but the conclusions hold: 40–45% of ranks change, and 7 of the 8 top-1 changes persist. Across the 69 direct pairs, the factors have 2–22 levels (median 4), and the difficulty proxy has at most four bins.

Small tasks are a limitation of the data. The 4 pairs that fail the per-pair test are on the two smallest tasks (N = 91 and 128), whose contingency tables are sparse, so their classifications are data-limited. Excluding these two tasks still leaves 7 top-1 changes and 41% of ranks changed. We now state in Section 3.2 that CST is a screening step at a fixed threshold, not a test with controlled error rates, and mark the one top-1 change on such a task (VSTI-Bench camera–object distance v1) in Section 4.3.


[W4] Reliability

We added per-task weight diagnostics on the full benchmarks to Appendix F (Table F.4): items, joint cells, ESS/n, the largest item weight, the weight on the top 5% of items and on cells with fewer than 5 items, and a 10× weight cap. All numbers use the revised factors (see our opening comment).

(1) “rare joint cells [...] can receive extreme weights.”

Extreme weights drive a top-1 change on one task only. Cells with fewer than 5 items hold over half of the weight on 3 of the 29 reweighted tasks. Capping every item weight at 10 times the uniform weight keeps the FBS winner on 7 of the 8 top-1 changes. The exception is VSI-Bench object counting, discussed below.

(2) “it is hard to judge how much of each benchmark effectively survives reweighting.”

Median ESS/n is 0.80; the one exception is VSI-Bench object counting (ESS/n = 0.04). Its only question with count 1 receives 113 times the uniform weight, and this single question drives the task's top-1 change: without it, ESS/n is 0.72, the top-ranked model does not change, and 4 instead of 13 model ranks change. On the other 28 tasks, ESS/n is at least 0.23, and 39.5% of model ranks and 7 of 28 top-1 models still change. We have updated the manuscript accordingly.

[W5] Scope beyond video benchmarks

Thank you for suggesting an out-of-family demonstration. We applied CST and FBS, with the same settings as in the paper, to three benchmarks outside the video family: CV-Bench (single-image), RoboSpatial (robotics-oriented) and SPAR-Bench (single- and multi-view; its eight numerical depth and distance tasks).

Table 3-3. CST and FBS on three additional benchmarks

BenchmarkRanks changedTop-1 changed
CV-Bench10 / 24 (41.7%)1 / 4
RoboSpatial19 / 36 (52.8%)2 / 3
SPAR-Bench29 / 88 (33.0%)0 / 8
All58 / 148 (39.2%)3 / 15

Composition-dependent re-ranking occurs in all three additional benchmarks, while its extent varies. CST identifies at least one direct contributor on 13 of the 15 tasks, and across these 58 of 148 model-task ranks change (39.2%). The top-1 model changes on 3 tasks: on CV-Bench counting from SpatioLM-InternVL to Qwen3-VL-8B (direct contributors: number of options, answer value and image source), and on RoboSpatial compatibility and context from Qwen3-VL-8B to InternVL3.5-8B (answer value for compatibility; scene complexity, brightness range and sharpness for context). On SPAR-Bench the top-1 model is unchanged on all eight tasks, while a third of the ranks below it change. Answer value is a direct contributor on every SPAR-Bench task.

Please note that our framework is not tied to a particular data format. These additional results demonstrate that composition sensitivity extends beyond the four video benchmarks studied in the original manuscript. We also added these results in our revised manuscript.

Dear Reviewer 1iqo,

We sincerely thank you for carefully considering our rebuttal. As we approach the end of the discussion period, please let us know whether further clarification would be helpful regarding the discussed points. Your feedback has been valuable in improving our paper.

Best regards,

Authors of Submission 15446

Reviewer_7J5ZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines how the composition of spatial reasoning benchmarks affects model scores and rankings. It proposes CST to identify factors associated with average model performance after adjustment for other factors. It then uses FBS to reduce the influence of frequently represented factor levels through reweighting. The analysis covers four benchmarks, 39 tasks, and 14 models. Among the 29 tasks selected for balancing, 53.4% of model-task ranks change, and the top model changes on 10 tasks.

Critical strengths
  1. The paper moves beyond individual annotation errors and studies benchmark composition. The analysis spans several datasets and models. Changes in task winners and specialized-model advantages show that this issue can affect research conclusions.

  2. The approach can reuse existing item-level predictions without new model inference. Reporting original scores, balanced scores, and composition advantages gives a more complete picture than a single average.

Critical weaknesses
  1. The object-type proxy depends on an evaluated model. Appendix D defines this factor using item-level scores from Qwen2.5-VL-7B. This is different from directly using object categories. Since this model is also evaluated, the factor construction may influence both diagnosis and reweighting. Sharing weights across models does not remove this dependence. An ablation excluding this proxy would help establish whether the main conclusions remain valid. The author should use a more powerful opensource model (like kimi-k3) as the proxy.

  2. Intrinsic difficulty and undesirable evaluation bias need clearer separation. Answer values can define genuine task difficulty. For example, counting 20 chairs may require more perception and memory than counting 2 chairs. How do you distinguish intrinsic task difficulty from an undesirable composition effect? What capability does equal weighting across answer-value bins measure, and why should this target complement the original distribution? A human study is not required to address this conceptual question.

  3. CST uses average scores across models. A factor could affect two models in opposite directions while leaving their average unchanged. CST may therefore miss some factors that matter for relative rankings. Single-factor adjustments also do not fully resolve joint dependence.

Questions for authors

Please see weaknesses.

4 comments

We sincerely thank you for the careful review, and for recognizing that our work “gives a more complete picture than a single average”.


[W1] Object-type proxy

Thank you for pointing this out and for the concrete suggestion. We agree that the former “object type” depended on an evaluated model, and the term was misleading as it depended on per-item scores. Following your suggestion and reviewer wt8n’s comment, we made the following two changes, re-ran the full CST → FBS pipeline on all 39 tasks, and revised the paper accordingly:

  • “Difficulty proxy” based on Kimi-K3. The former “object type” was indeed more of a per-item difficulty measure than an object category. As you suggested, we now compute it with Kimi-K3, which is not among the evaluated models, and rename it “difficulty proxy”.
  • A new “object category” factor. As you note, the former factor differs from directly using object categories. We added a new factor, “object category”, derived by extracting the objects in the text query of each question and grouping them into semantic categories.

The Qwen2.5-VL-7B-based proxy is no longer used anywhere in the pipeline, so the revised results also show whether the main conclusions hold without it. They do, as shown in Table 2-1: balancing changes the top-ranked model on 8 of 29 balanced tasks (previously 10) and 168 of 406 model-task ranks (41.4%, previously 53.4%), and it still reduces score variability by 59-69% under composition resampling. All revised text, tables, and figures are highlighted in color in the revised PDF, and we would be grateful if you could take a look at these parts. The two factors are defined in Section 3.1 and Appendix D.

Table 2-1. Overall results before and after the revision, over the 29 tasks balanced by FBS.

BeforeAfter
Tasks with top-1 changed10 / 298 / 29
Model-task ranks changed217 / 406 (53.4%)168 / 406 (41.4%)

Table 2-2 shows how often each revised factor is diagnosed as a direct contributor. The “difficulty proxy” is a direct contributor on 11 tasks (previously 17), and the newly added “object category” on 6 tasks.

Table 2-2. Number of tasks where each factor is a direct contributor, over all 39 tasks.

FactorBeforeAfter
“object type” (Qwen2.5-VL-7B)“difficulty proxy” (Kimi-K3)“object category”
Tasks as direct contributor17 / 3911 / 396 / 39

Finally, the revision removed the dependence you pointed out. Because the former difficulty measure was computed from Qwen2.5-VL-7B's own per-item scores, balancing sometimes favored this model: on the 17 tasks where it was a direct contributor, Qwen2.5-VL-7B gained +0.113 on average against +0.013 for the other models. With the Kimi-K3-based proxy, its gain (+0.008) is in line with the others (−0.017). We thank the reviewer again for this suggestion, which made our analysis fairer and our conclusions more reliable.


[W2] Role of equal weighting

Thank you for this question. We first refer the reviewer to Appendix A, where we discuss this point (“Does FBS provide universally more faithful model comparisons?” and “Why use a uniform reference for FBS?”). Please note that we do not claim that FBS universally yields a more faithful ranking, nor that equal weight is the “correct” target. Our motivation is the composition bias introduced when benchmark data are acquired, which the reviewer also recognizes as a problem that can affect research conclusions. We summarize the discussion below.

“Difficulty” describes how hard each factor level is, whereas “composition effect” describes how the benchmark’s mix of factor levels decides which model wins. We agree that intrinsic difficulty is related to evaluation bias. Yet even for an obvious difficulty such as counting, models can differ in which counts they handle well. So the benchmark’s composition, whether it contains mostly small or mostly large counts, can decide which model ranks higher. For example, on OSI-Bench depth-aware counting, SpatioLM-InternVL ranks first overall, ahead of SenseNova-SI-8B (0.42 vs. 0.39), because it is much better on counts of 0-1 (0.61 vs. 0.33), which make up half of the questions, while SenseNova-SI-8B is better on counts of 2-3. FBS ranks SenseNova-SI-8B first. Please note that our work focuses on such composition effects: FBS keeps every per-item score and changes only how much each level counts.

Choosing the right target remains an open question. A universal “real-world” distribution is difficult to define, since spatial quantities vary substantially across environments, datasets, and deployment settings: mean object size ranges from 13 to 390 cm and mean object count from about 1-2 to 56 across the datasets in Table A.1. For example, 75% of the object size values in ReVSI lie between 25 and 150 cm, while the mean object size is 13 cm for household objects in YCB and 390 cm in KITTI driving scenes (Table A.1). Choosing any single dataset distribution as the reference would therefore privilege a particular environment or data-collection process. In the absence of a justified external target, we use a uniform target as a transparent, frequency-neutral reference, which stays within the support of the benchmark. How to choose a reliable target distribution is a crucial question for benchmarking that calls for future research, and when a task-specific target is available, our framework can be used to balance toward it.


[W3] Average-based selection criterion of CST

Thank you for raising this concern. We clarify the motivation for using the model-averaged score, assess how re-ranking findings change under individual-model scores, examine factor coverage and the effects of top-ten screening, and discuss joint dependence. All results in this response use the revised experimental setup described in our response to W1, with the difficulty proxy computed from Kimi-K3 and the new object-category factor.

(1) Why use the average score across models?

Our choice of the per-item mean score is motivated by CST's objective of diagnosing benchmark-level composition sensitivity. While this diagnosis necessarily relies on model outputs, we aim to identify direct contributors using a reference that represents the full evaluated model pool, rather than the behavior of any individual model. Averaging across all models gives each model an equal weight in shaping the diagnosis result, providing a common reference for factor selection. Moreover, this model-neutral design of CST further allows us to analyze relative model rankings through FBS.

(2) Does using individual models instead of the mean change our conclusions?

As the reviewer points out, averaging can hide factors that affect models in opposite directions. To assess sensitivity to the reference score, we re-ran CST using each model's individual scores, keeping factor preprocessing, CST hyperparameters, and the FBS procedure unchanged. Each run still produces one shared weight vector per task for evaluating all 14 models.

Table 2-3. Effect of the reference score used for CST on FBS rankings.

CST referenceTasks reweightedTop-1 changesRanks changed (%)Top-1 changes preserved
Mean of 14 models (paper)29841.48 / 8
Individual model (min–max)28–375–1036.7–46.34–7 / 8

The main re-ranking findings are broadly preserved when CST uses individual-model scores instead of the mean score. Across all models, 4–7 of the original eight top-1 changes are preserved, averaging 5.6.

The overall extent of re-ranking is also comparable: individual-model CST produces 5–10 top-1 changes, compared to 8 in the original setting. Together, these results show that individual-model diagnostics largely preserve the original re-ranking findings, aligning with our core conclusion that benchmark composition substantially affects model comparisons, even when CST does not rely on the mean score.

(3) Factor coverage

To measure how often averaging misses factors that matter for individual models, we also compare the direct contributors found by CST on each model's score with those found by CST on the average score (Table 2-4).

Table 2-4. Direct contributors from per-model CST, and whether the original CST also selects them.

Models selecting the pairFactor–task pairsSelected by original CST
≥ 119966 (33%)
≥ 48259 (72%)
≥ 74843 (90%)
≥ 103027 (90%)
All 141514 (93%)

Average-based CST captures the factors shared across models. It selects 90% of the factor–task pairs that at least half of the 14 models select, and 93% of the pairs that all 14 models select. The misses concentrate on pairs selected by only a few models. This follows the purpose of the mean score in (1): CST diagnoses composition sensitivity shared across the evaluated model pool, rather than patterns specific to one or two models. As shown in (2), running CST on individual models instead does not change our main conclusions.

(4) Joint dependence

We would like to draw the reviewer's attention to Appendix E (“Robustness to joint conditioning”) and Table E.2, where we repeat the diagnosis while conditioning jointly on two and three factors. Of the 69 direct factor–task pairs, 58 (84%) remain direct under two-factor conditioning, and none falls below the magnitude threshold: the 11 changes are reclassifications as suppressed or surface correlates. Under three-factor conditioning, 33 (48%) remain direct, but the mean number of occupied conditioning strata grows from 4.6 to 51.7, so the estimates become increasingly data-limited. We agree that single-factor adjustment does not fully resolve joint dependence; we use it for the main diagnosis and treat joint conditioning as a sensitivity analysis, and larger benchmarks would enable more reliable multi-factor conditioning.

Dear Reviewer 7J5Z,

We sincerely thank you for carefully considering our rebuttal. As we approach the end of the discussion period, please let us know whether further clarification would be helpful regarding the discussed points. Your feedback has been valuable in improving our paper.

Best regards,

Authors of Submission 15446

Reviewer_wt8nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the sample composition of video spatial reasoning benchmarks influences model scores and rankings. The authors evaluate 14 vision-language models across 39 tasks on the VSI-Bench, ReVSI, VSTI-Bench and OSI-Bench datasets, proposing the Composition Sensitivity Test (CST) to identify factors correlated with task-specific scores, and subsequently applying the Factor-Balanced Score (FBS) to approximate a balanced weighting of the observed values of these factors. The paper reports that, across the 29 tasks where relevant factors were identified, the model–task rankings changed for 217 out of 406 models, with the top-ranked model changing for 10 tasks; in resampling experiments where the sample composition was altered according to the selected factors, the FBS reduced score fluctuations across compositions by 63 per cent to 71 per cent. These results indicate that raw overall scores may be significantly influenced by the distribution of the benchmark samples.

Critical strengths
  1. I find the empirical problem important and convincingly demonstrated. Across four benchmarks and 14 VLMs, the authors show that changing only the aggregation weights of existing predictions can substantially alter model comparisons. The top-ranked model changes on 10 of the 29 reweighted tasks. This makes benchmark composition a consequential issue for interpreting spatial reasoning results.

  2. I value the practical framework for making composition effects visible. CST identifies candidate factors, while FBS and composition advantage quantify how the reported scores depend on their frequencies. The analyses of alternative weighting schemes and resampling settings add useful context. I particularly appreciate the recommendation to report observed and balanced scores together.

Critical weaknesses
  1. I am not convinced that the factor-balanced score supports the paper’s stronger interpretations of model rankings.

    Ground-truth answer value is one of the main factors being balanced, yet in counting and metric estimation it can reflect the difficulty of the spatial task itself. Giving its observed levels approximately equal weight therefore changes the evaluation target; the paper does not establish why that target is more appropriate than the benchmark’s original distribution. The reported rank reversals convincingly demonstrate sensitivity to aggregation choices, but, without an independently justified target distribution or validation criterion, I cannot tell whether the balanced rankings provide more meaningful comparisons of spatial reasoning ability.

  2. I am concerned that CST’s selection criterion is not well aligned with the paper’s central concern: changes in model rankings. CST ranks candidate factors by mutual information with the per-item score averaged across all 14 models, then retains only the top ten. A factor can have little association with this average while strongly affecting relative model performance: if two models perform well on opposite factor levels, their scores may cancel in the average even though changing the levels’ frequencies reverses their ranking. The paper does not establish that CST retains such factors, or show that its factor selection leads to more informative ranking analyses than simpler alternatives. This uncertainty matters because every subsequent FBS result depends on which factors CST selects.

Questions for authors
  1. Evaluation target

    What interpretation do you intend for FBS when it gives approximately equal weight to observed answer-value levels, especially in counting and metric estimation tasks where answer value may reflect task difficulty? Do the headline ranking reversals persist under other plausible target distributions? This would help me assess whether the reversals reveal a limitation of the original benchmark or sensitivity to a particular alternative target.

  2. CST’s alignment with ranking sensitivity

    Why is mutual information with the average per-item score across models an appropriate criterion for selecting factors that affect relative model rankings? Opposing model-performance patterns can cancel in that average while producing a ranking reversal under a composition shift. How often does CST miss factors identified by a direct measure of model-performance heterogeneity or ranking sensitivity, and how does the top-ten screening step affect that result?

  3. Definition of “object type”

    Appendix D defines this factor using the per-item score of Qwen2.5-VL-7B, which is also among the evaluated models. How should readers interpret it as an object-category factor? Do the CST diagnoses and principal FBS ranking changes persist when it is replaced with actual object-category metadata or a difficulty measure independent of the evaluated models?

  4. Reliability of the headline reversals

    For the ten tasks whose top-ranked model changes, what are the before-and-after score gaps and their uncertainty intervals when resampling by video or scene? How many reversals remain when CST factors are reselected within each resample? These results would help determine whether the reported reversals are robust to both item sampling and factor selection.

6 comments

We sincerely thank you for the thoughtful review, and for finding the problem “important and convincingly demonstrated” and our framework “practical” for “making composition effects visible”.


[Q3] Definition of “object type”

Thank you for pointing this out. Your suggestion helped us refine its definition and strengthen our analysis. We agree that “object type” was a misleading term and biased because it was based on Qwen2.5-VL-7B. We made the following two changes, re-ran the full CST → FBS pipeline on all 39 tasks, and revised the paper accordingly:

  • “Difficulty proxy” based on Kimi-K3. The former “object type” was indeed more of a per-item difficulty measure computed from Qwen2.5-VL-7B, one of the evaluated models. We rename this factor as “difficulty proxy” and compute it with Kimi-K3 instead, which is not among the evaluated models.
  • A new “object category” factor. We added a new factor, “object category”, derived by extracting the objects in the text queries in each question and grouped into semantic categories. This acts as the “object-category metadata” that the reviewer suggested.

With the revised factors, the main conclusions still hold as shown in Table 1-1. Balancing changes the top-ranked model on 8 of 29 balanced tasks (previously 10) and 168 of 406 model-task ranks (41.4%, previously 53.4%), and it still reduces score variability by 59-69% under composition resampling. All revised text, tables, and figures are highlighted in color in the revised PDF, and we would be grateful if you could take a look at these parts. The two factors are defined in Section 3.1 and Appendix D.

Table 1-1. Overall results before and after the revision, over the 29 tasks balanced by FBS.

BeforeAfter
Tasks with top-1 changed10 / 298 / 29
Model-task ranks changed217 / 406 (53.4%)168 / 406 (41.4%)

Table 1-2 shows how often each clarified factor is diagnosed as a direct contributor. The “difficulty proxy” is a direct contributor on 11 tasks (previously 17), and the newly added “object category” on 6 tasks.

Table 1-2. Number of tasks where each factor is a direct contributor, over all 39 tasks.

FactorBeforeAfter
“object type” (Qwen2.5-VL-7B)“difficulty proxy” (Kimi-K3)“object category”
Tasks as direct contributor17 / 3911 / 396 / 39

Finally, the revision removed the dependence on Qwen2.5-VL-7B. Because the former difficulty measure was from its own per-item scores, balancing sometimes favored this model: on the 17 tasks where the difficulty proxy was a direct contributor, Qwen2.5-VL-7B gained +0.113 on average against +0.013 for the other models. With the modified Kimi-K3-based proxy, its gain (+0.008) is in line with the others (−0.017). We thank the reviewer again for this suggestion, which made our analysis fairer and our conclusions more reliable.


[W1, Q1] Evaluation target of FBS

(1) “The paper does not establish why that target is more appropriate than the benchmark’s original distribution.”

We first refer the reviewer to Appendix A, where we discuss this point (“Does FBS provide universally more faithful model comparisons?” and “Why use a uniform reference for FBS?”). Please note that we do not claim that FBS universally yields a more faithful ranking, nor that equal weight is the “correct” target. Our motivation is the composition bias introduced when benchmark data are acquired, which the reviewer also sees as “a consequential issue”. We agree that answer values can be correlated with genuine task difficulty, and we do not aim to remove this difficulty. We summarize the discussion below.

Choosing the right target remains an open question. A universal “real-world” distribution is difficult to define, since spatial quantities vary substantially across environments, datasets, and deployment settings: mean object size ranges from 13 to 390 cm and mean object count from about 1-2 to 56 across the datasets in Table A.1. For example, 75% of the object size values in ReVSI lie between 25 and 150 cm, while the mean object size is 13 cm for household objects in YCB and 390 cm in KITTI driving scenes (Table A.1). Choosing any single dataset distribution as the reference would therefore privilege a particular environment or data-collection process. In the absence of a justified external target, we use a uniform target as a transparent, frequency-neutral reference, which stays within the support of the benchmark. How to choose a reliable target distribution is a crucial question for benchmarking that calls for future research, and when a task-specific target is available, our framework can be used to balance toward it.

(2) “Do the headline ranking reversals persist under other plausible target distributions?”

We also tested other plausible target distributions. As targets, we took the mean values of the real data corpora listed in Table A.1 (benchmarks excluded). Since only the means are reported, we rescale each task’s answer distribution to the corpus mean, keeping its shape and coefficient of variation fixed. We apply this to every object counting and object size task where the answer value is a direct contributor (four counting tasks and ReVSI object size). We use the revised factors (see [Q3] above). As shown in Table 1-3, re-ranking and top-1 changes persist under different targets (72% of model ranks; top-1 in 15 of 19 task-target combinations).

Table 1-3. Rankings under answer-value targets from Table A.1, compared with the observed ranking.

TargetMeanModel ranks changedTop-1 changed
Object count (4 tasks)
Uniform (FBS)–41 / 56 (73%)2 / 4
Indoor scan datasets (low end)1.347 / 56 (84%)3 / 4
Indoor scan datasets (high end)2.150 / 56 (89%)4 / 4
FSC-1475647 / 56 (84%)2 / 4
Object size (1 task)
Uniform (FBS)–5 / 14 (36%)1 / 1
YCB13 cm11 / 14 (79%)1 / 1
MultiScan74 cm7 / 14 (50%)1 / 1
ScanNet++80 cm5 / 14 (36%)1 / 1
ARKitScenes87 cm5 / 14 (36%)1 / 1
3RScan96 cm5 / 14 (36%)1 / 1
ScanNet98 cm5 / 14 (36%)1 / 1
KITTI390 cm9 / 14 (64%)0 / 1
Total (corpus targets)191 / 266 (72%)15 / 19

[W2, Q2] Average-based selection criterion of CST

We clarify the motivation for using the model-averaged score, assess how re-ranking findings change under individual-model scores, and examine factor coverage and the effects of top-ten screening. Results are based on the refined setup in [Q3].

(1) Why use the average score across models?

Our choice of the per-item mean score is motivated by CST's objective of diagnosing benchmark-level composition sensitivity. While this diagnosis necessarily relies on model outputs, we aim to identify direct contributors using a reference that represents the full evaluated model pool, rather than the behavior of any individual model. Averaging across all models gives each model an equal weight in shaping the diagnosis result, providing a common reference for factor selection. Moreover, this model-neutral design of CST further allows us to analyze relative model rankings through FBS.

(2) Does using individual models instead of the mean change our conclusions?

We acknowledge that averaging can obscure opposing model-specific dependencies. To assess sensitivity to the reference score, we re-ran CST using each model's individual scores, keeping factor preprocessing, CST hyperparameters, and the FBS procedure unchanged. Each run still produces one shared weight vector per task for evaluating all 14 models.

Table 1-4. Effect of the reference score used for CST on FBS rankings.

CST referenceTasks reweightedTop-1 changesRanks changed (%)Top-1 changes preserved
Mean of 14 models (paper)29841.48 / 8
Individual model (min-max)28-375-1036.7-46.34-7 / 8

The main re-ranking findings are broadly preserved when CST uses individual-model scores instead of the mean score. Across all models, 4-7 of the original eight top-1 changes are preserved, averaging 5.6.

The overall extent of re-ranking is also comparable: individual-model CST produces 5-10 top-1 changes, compared to 8 in the original setting. Together, these results show that individual-model diagnostics largely preserve the original re-ranking findings, aligning with our core conclusion that benchmark composition substantially affects model comparisons, even when CST does not rely on the mean score.

(3) Factor coverage and the effect of top-10 screening

To measure the effect of model-performance heterogeneity directly, we also compare the direct contributors found by CST on each model's score with those found by CST on the average score (Table 1-5).

Table 1-5. Direct contributors from per-model CST, and whether the original CST also selects them.

Models selecting the pairFactor-task pairsSelected by original CSTRemoved by top-10 screen
≥ 119966 (33%)26
≥ 48259 (72%)2
≥ 74843 (90%)0
≥ 103027 (90%)0
All 141514 (93%)0

Average-based CST captures the factors shared across models. It selects 90% of the factor-task pairs that at least half of the 14 models select, and 93% of the pairs that all 14 models select. The misses concentrate on pairs selected by only a few models. This follows the purpose of the mean score in (1): CST diagnoses composition sensitivity shared across the evaluated model pool, rather than patterns specific to one or two models. As shown in (2), running CST on individual models instead does not change our main conclusions.

The top-10 screen does not drive the misses. Without the screen, CST on the average score selects no factor outside the top ten, and the FBS results are nearly unchanged (8 top-1 changes; 40.9% vs. 41.4% of ranks changed). The screen also removes none of the 48 pairs selected by at least half of the models.


[Q4] Reliability of the headline reversals

We thank the reviewer for this suggestion. With the revised factors (see Q3), 8 tasks have a top-1 change; we report these eight (Table 1-6; details in the new Appendix H).

Procedure. For each task, A and B are the observed and the FBS top-1 models on the full data, and we track this same pair in every resample. We resample by scene or video: half of a task's scenes or videos are drawn without replacement, 1,000 times, keeping all questions of a scene together and scoring all 14 models on the same subsample (for a mean, the spread over half-samples matches the sampling spread of the full-data estimate). A reversal is retained when A scores at least as high as B before FBS and B scores higher after FBS. With factors fixed, the CST factors and bins selected on the full data are kept and the FBS weights are re-solved on each subsample; with factors reselected, CST and FBS are re-run on each subsample.

Results. On OSI-Bench depth-aware counting, the example in our abstract and Fig. 1, the gap Δ = Score(B) − Score(A) moves from −2.9 pp [−6.5, +0.8] before FBS to +7.9 pp [+4.0, +12.2] after, and the reversal is retained in 93% of the subsamples with factors fixed and in 92% with factors reselected. Over the eight tasks, 4.5 reversals are retained per subsample with factors fixed (95% range 2–7) and 3.6 with factors reselected (1–6), so reselecting the factors costs less than one reversal on average. Table 1-6 gives the intervals and retention for every task.

Table 1-6. Score gaps Δ (pp) with 95% intervals, and reversal retention, over 1,000 half-samples.

TaskBefore FBSAfter FBSRetained, factors fixedRetained, factors reselected
OSI-Bench depth-aware counting−2.9 [−6.5, +0.8]+7.9 [+4.0, +12.2]93%92%
OSI-Bench ego-motion−6.1 [−8.4, −3.4]+1.5 [−3.3, +6.6]74%71%
VSTI-Bench cam–obj distance v2−3.2 [−9.4, +2.6]+4.3 [−3.8, +11.5]70%35%
VSI-Bench object counting−6.5 [−9.4, −3.2]+13.9 [−12.0, +16.9]51%42%
OSI-Bench absolute distance−1.0 [−3.9, +2.1]+1.1 [−1.3, +4.1]54%51%
VSI-Bench room size−1.4 [−4.0, +1.3]+1.4 [−3.8, +5.3]48%27%
ReVSI object size−0.3 [−1.8, +1.0]+0.3 [−1.5, +2.1]35%32%
VSTI-Bench cam–obj distance v10.0 [−6.4, +6.2]+3.3 [−5.3, +8.4]24%8%
Reversals retained per subsample (of 8)4.5 (2–7)3.6 (1–6)

Interpretation. Half-sampling keeps each benchmark's composition in expectation, so it measures how much a reversal depends on which items are sampled. FBS up-weights rare conditions, so its gaps vary more under this resampling (wider intervals after FBS on 7 of the 8 tasks), and reversals between closely scored models stay within this sampling uncertainty. The widest interval, on VSI-Bench object counting, comes from a single question that carries 20% of the weight (App. F); we no longer use that task as a headline example. Robustness to composition shifts, which FBS targets, is tested in Section 4.3 and App. G (mean Kendall's τ rises from 0.56–0.81 with observed scores to 0.82–0.86 with FBS across the four benchmarks) and, for shifts in directions CST does not choose, in the new App. I (0.74 → 0.80).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.