- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how the composition of spatial reasoning benchmarks—distributions of task-agnostic factors such as ground-truth answer value, queried object type, room type, or correct option—shapes aggregate scores and leaderboard rankings. The aggregate score is decomposed as a frequency-weighted average over factor configurations (Eq. 2). Two tools are introduced: (1) the Composition Sensitivity Test (CST), which uses marginal vs. adjusted (conditional) mutual information between discretized per-item scores and candidate factors, combined with a magnitude threshold and a ratio band, to classify factors as direct contributors / suppressed / surface correlates / negligible; and (2) the Factor-Balanced Score (FBS), which reweights items via stable balancing weights so that diagnosed direct contributors have approximately uniform marginals, yielding a balanced score and a per-model "composition advantage" (observed − balanced). Across 4 video spatial benchmarks, 39 tasks, and 14 VLMs, CST flags answer value (26/39 tasks) and object type (17/39) as the dominant contributors; balancing changes 53.4% of model-task ranks and the top-1 model on 10/29 balanced tasks; spatially fine-tuned models often fall below their own backbones after balancing; and FBS reduces score variability by 63–71% under systematic composition resampling.
Critical strengths
- An important and understudied validity threat, crisply formalized. The observation that benchmark scores are composition-weighted averages—and that skewed answer-value/object-type distributions can decide leaderboard winners—is made concrete with a clean decomposition (Eq. 2) and striking worked examples (e.g., VSI-Bench counting, where balancing moves Qwen3-VL-8B from eighth to first and shrinks SpatialLadder's lead over Qwen2.5-VL-3B from 0.38 to 0.03).
- Methodologically principled. Using CMI to adjust for factor entanglement addresses the obvious objection that factor–score associations are confounded; using stable balancing weights (Zubizarreta, 2015) rather than naive inverse-frequency weighting controls weight dispersion. The association-not-causation caveat for "direct contributor" is stated honestly.
- Scale and consistency of the empirical evidence. Four benchmarks, 39 tasks, 14 models, with per-regime diagnosis (Fig. 3), full re-ranking statistics, composition-advantage heatmaps, and a systematic resampling protocol. The finding that relational tasks have no direct contributor while counting/quantitative tasks are dominated by answer value is a useful, actionable map for benchmark designers.
- Actionable recommendation. Reporting (observed, balanced, composition advantage) jointly is cheap to adopt and immediately interpretable; the CA metric nicely exposes who benefits from a benchmark's quirks (e.g., spatially fine-tuned models gaining +0.16 to +0.25 on VSI-Bench counting purely from composition).
Critical weaknesses
- The robustness evaluation is partly circular. The composition-resampling experiment perturbs compositions along the top-1 direct contributor and shows FBS scores/rankings vary less. But FBS is constructed to flatten exactly that factor's marginal influence, so reduced sensitivity to resampling along it is close to guaranteed by design. A more convincing robustness test would perturb along factors not balanced (e.g., the second-ranked contributor, or held-out factors like brightness), or subsample items uniformly at random (which changes all factor marginals jointly) and check whether FBS-on-subset A agrees with FBS-on-subset B more than observed scores do.
- The uniform-marginal target is doing normative work that the paper under-interrogates. FBS answers "what would the ranking be if factor levels were equally represented?"—not "what is the true ranking." The headline re-ranking results (53.4% of ranks, 10/29 winners) are rhetorically framed as exposing misleading leaderboards, but an equally valid reading is that the balanced ranking is simply a different ranking whose target distribution is no more grounded in deployment reality. The limitation section acknowledges this, but the abstract/main text ("unintentionally favor", "alleviate the dominance") leans toward treating balanced as better. For counting tasks in particular, low-count dominance may reflect the real frequency of objects in scenes, not a construction artifact.
- MI/CMI estimation on small, discretized samples is fragile and not stress-tested. Per-task item counts range from ~100–900; Y is discretized into 10 equal-width bins; factors like object type have many levels. Empirical MI is notoriously biased upward at these sample sizes, and CMI even more so. The sensitivity analysis covers the threshold constant c (App. E), but I could not find permutation-based null distributions or bias-corrected MI estimators—without them, the direct/suppressed/negligible classifications may inherit estimation noise.
- Marginal balancing ignores joint configurations. FBS equalizes each diagnosed factor's marginal, but entangled factors mean rare joint cells (e.g., a specific room type × answer value) can receive extreme weights. SBW limits dispersion, but no effective sample size or weight-distribution statistics are reported per task, so it is hard to judge how much of each benchmark effectively survives reweighting.
- Scope is narrow relative to the framing. All four benchmarks are recent video spatial-reasoning suites with similar construction philosophies. Claims about "spatial reasoning benchmarks" generally—and the recommendation's portability to e.g. image-based or embodied benchmarks—would benefit from at least one out-of-family demonstration.
Questions for authors
See weaknesses above
4 comments
(2) “For counting tasks in particular, low-count dominance may reflect the real frequency of objects in scenes, not a construction artifact.”
We tested real-frequency targets directly. We used the mean values of the real data corpora in Table A.1 as answer-value targets, rescaling each task’s answer distribution to the corpus mean with its shape and coefficient of variation fixed (revised factors; see our opening comment). First, the four counting tasks are less concentrated on low counts than real indoor scenes (mean answer 2.6–7.0 vs. 1.3–2.1), so their composition is not a direct record of real object frequencies. Second, re-ranking under real-frequency targets is even larger than under the uniform target (84–89% vs. 73% of model ranks; Table 3-2). Including object size, 72% of model ranks and the top-1 model in 15 of 19 task–target combinations change, so the reversals are not specific to the uniform target.
Table 3-2. Object counting rankings (4 tasks, 14 models) under answer-value targets from Table A.1, compared with the observed ranking.
| Target | Mean | Model ranks changed | Top-1 changed |
|---|---|---|---|
| Uniform (FBS) | – | 41 / 56 (73%) | 2 / 4 |
| Indoor scan datasets (low end) | 1.3 | 47 / 56 (84%) | 3 / 4 |
| Indoor scan datasets (high end) | 2.1 | 50 / 56 (89%) | 4 / 4 |
| FSC-147 | 56 | 47 / 56 (84%) | 2 / 4 |
(3) “The abstract/main text […] leans toward treating balanced as better.”
We have revised the wording accordingly, presenting the changed ranks (41.4%) and winners (8/29) as the sensitivity of comparisons to composition, not as leaderboard errors:
- “alleviate the dominance of frequently represented levels of diagnosed factors” → “make the observed levels of diagnosed factors contribute equally”.
- “can unintentionally favor some models over others” → “can decide which model ranks first”.
[W3] Reliability of the MI/CMI estimates
Bias correction and permutation nulls are part of CST. MI and CMI are estimated with the Miller-Madow correction (within each stratum for CMI), and c is calibrated against a permutation null (200 score shuffles per task): at c = 0.02, the resulting threshold γₜ = cH(Yₜ) exceeds the 95th percentile of the chance MI of every direct contributor on all tasks with N ≥ 200. We have made this explicit in Section 3.2. We also tested each of the 69 direct factor–task pairs against a per-pair permutation null for MI and a within-strata null for CMI: 65 pass both (median ratio to the 95th percentile: 7.3 for MI, 5.6 for CMI).
Discretization. With 5 or 3 score bins instead of 10, the selected factor sets change on about half of the tasks, but the conclusions hold: 40–45% of ranks change, and 7 of the 8 top-1 changes persist. Across the 69 direct pairs, the factors have 2–22 levels (median 4), and the difficulty proxy has at most four bins.
Small tasks are a limitation of the data. The 4 pairs that fail the per-pair test are on the two smallest tasks (N = 91 and 128), whose contingency tables are sparse, so their classifications are data-limited. Excluding these two tasks still leaves 7 top-1 changes and 41% of ranks changed. We now state in Section 3.2 that CST is a screening step at a fixed threshold, not a test with controlled error rates, and mark the one top-1 change on such a task (VSTI-Bench camera–object distance v1) in Section 4.3.
[W4] Reliability
We added per-task weight diagnostics on the full benchmarks to Appendix F (Table F.4): items, joint cells, ESS/n, the largest item weight, the weight on the top 5% of items and on cells with fewer than 5 items, and a 10× weight cap. All numbers use the revised factors (see our opening comment).
(1) “rare joint cells [...] can receive extreme weights.”
Extreme weights drive a top-1 change on one task only. Cells with fewer than 5 items hold over half of the weight on 3 of the 29 reweighted tasks. Capping every item weight at 10 times the uniform weight keeps the FBS winner on 7 of the 8 top-1 changes. The exception is VSI-Bench object counting, discussed below.
(2) “it is hard to judge how much of each benchmark effectively survives reweighting.”
Median ESS/n is 0.80; the one exception is VSI-Bench object counting (ESS/n = 0.04). Its only question with count 1 receives 113 times the uniform weight, and this single question drives the task's top-1 change: without it, ESS/n is 0.72, the top-ranked model does not change, and 4 instead of 13 model ranks change. On the other 28 tasks, ESS/n is at least 0.23, and 39.5% of model ranks and 7 of 28 top-1 models still change. We have updated the manuscript accordingly.
[W5] Scope beyond video benchmarks
Thank you for suggesting an out-of-family demonstration. We applied CST and FBS, with the same settings as in the paper, to three benchmarks outside the video family: CV-Bench (single-image), RoboSpatial (robotics-oriented) and SPAR-Bench (single- and multi-view; its eight numerical depth and distance tasks).
Table 3-3. CST and FBS on three additional benchmarks
| Benchmark | Ranks changed | Top-1 changed |
|---|---|---|
| CV-Bench | 10 / 24 (41.7%) | 1 / 4 |
| RoboSpatial | 19 / 36 (52.8%) | 2 / 3 |
| SPAR-Bench | 29 / 88 (33.0%) | 0 / 8 |
| All | 58 / 148 (39.2%) | 3 / 15 |
Composition-dependent re-ranking occurs in all three additional benchmarks, while its extent varies. CST identifies at least one direct contributor on 13 of the 15 tasks, and across these 58 of 148 model-task ranks change (39.2%). The top-1 model changes on 3 tasks: on CV-Bench counting from SpatioLM-InternVL to Qwen3-VL-8B (direct contributors: number of options, answer value and image source), and on RoboSpatial compatibility and context from Qwen3-VL-8B to InternVL3.5-8B (answer value for compatibility; scene complexity, brightness range and sharpness for context). On SPAR-Bench the top-1 model is unchanged on all eight tasks, while a third of the ranks below it change. Answer value is a direct contributor on every SPAR-Bench task.
Please note that our framework is not tied to a particular data format. These additional results demonstrate that composition sensitivity extends beyond the four video benchmarks studied in the original manuscript. We also added these results in our revised manuscript.
Dear Reviewer 1iqo,
We sincerely thank you for carefully considering our rebuttal. As we approach the end of the discussion period, please let us know whether further clarification would be helpful regarding the discussed points. Your feedback has been valuable in improving our paper.
Best regards,
Authors of Submission 15446
We sincerely thank you for the thorough and constructive review, and for recognizing that we address “an important and understudied validity threat, crisply formalized” with a “methodologically principled” approach.
Revision with updated factor definitions. Based on the thoughtful suggestions from Reviewers wt8n and 7J5Z on more reliable factor definitions, we made the following two changes, re-ran the full CST → FBS pipeline on all 39 tasks, and revised the paper accordingly:
This makes our analysis fairer and more reliable, as it removes a possible bias from computing the difficulty proxy (the former “object type”) with Qwen2.5-VL-7B, one of the evaluated models. With the revised factors, the main conclusions still hold: balancing changes the top-ranked model on 8 of 29 tasks (previously 10) and 41.4% of model-task ranks (previously 53.4%). All revised text, tables, and figures are highlighted in color in the revised PDF, and we would be grateful if you could take a look at these parts. The two factors are defined in Section 3.1 and Appendix D.
[W1] Robustness evaluation
Thank you for this suggestion. We ran both suggested tests and one additional test (Table 3-1); their protocols and further variants are in the new Appendix I.
Table 3-1. Ranking agreement between two subsets (mean Kendall's τ of the 14-model rankings).
Random halves measure sampling noise, not composition. A random split keeps the distribution of every factor the same in both halves in expectation (mean change of a factor's marginal: 0.07, vs. 0.22 along the balanced factor in Section 4.3). The two rankings then differ only in which items are drawn, and FBS pays the variance cost of reweighting (0.76 → 0.69). The problem we address is the opposite case: subsets whose factor distributions differ.
We therefore also shift composition along random directions. Instead of the top-1 factor, each composition is tilted along a random linear combination of all top-10 candidate factors, redrawn every time, so all factor distributions change jointly, in directions CST does not choose. FBS raises agreement from 0.74 to 0.80 (higher on 24 of 29 tasks) and halves the score SD (0.045 → 0.023). The gain comes from the diagnosed factors: along unbalanced factors alone, FBS neither helps nor hurts (0.82 → 0.83 above; 0.79 → 0.80 in Appendix I).
[W2] Uniform-marginal target
(1) “An equally valid reading is that the balanced ranking is simply a different ranking whose target distribution is no more grounded in deployment reality.”
We agree, and first refer the reviewer to Appendix A (“Does FBS provide universally more faithful model comparisons?” and “Why use a uniform reference for FBS?”). FBS answers “what would the ranking be if factor levels were equally represented?”, not “what is the true ranking”, and we do not claim that equal weight is the “correct” target. Our motivation is the composition bias introduced when benchmark data are acquired.
Choosing the right target remains an open question. Spatial quantities vary widely across environments (Table A.1: mean object size 13–390 cm, mean object count about 1–2 to 56), so any single dataset distribution would privilege one environment. For example, 75% of ReVSI object sizes lie between 25 and 150 cm, while the mean object size is 13 cm in YCB and 390 cm in KITTI (Table A.1). Without a justified external target, we use the uniform target as a transparent, frequency-neutral reference within the benchmark’s support; when a task-specific target is available, our framework can balance toward it.