- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents FrontierSpatial, a benchmark and evaluation framework for visual spatial reasoning in multimodal language models. The manuscript reports 7,838 questions paired with 11,825 images, drawn from 23 existing datasets and organized into seven categories, 19 subcategories, and 65 fine-grained tasks. Its four-stage curation pipeline combines structural filtering, visual deduplication, text-based quality assessment, and multimodal validation.
The authors also propose an automated evaluation and error-analysis pipeline. It scores final-answer correctness and several spatial reasoning dimensions, clusters failed responses, and maps the resulting clusters to a hierarchical error taxonomy. The reported evaluation covers 16 models and a human baseline, emphasizing a gap between identifying spatial information and composing it into correct reasoning. Existing agentic systems are additionally evaluated on a small hard-case subset. The intended contribution is an integrated measurement and diagnostic framework, rather than a new spatial-reasoning model or agent architecture.
Critical strengths
-
A practically useful attempt to unify heterogeneous evaluation resources.
Bringing physical, geometric, relational, temporal, quantitative, perceptual, and knowledge-grounded tasks into a common interface could reduce the effort required to compare spatial capabilities across datasets. An integrated measurement resource can be valuable even without being the first comprehensive benchmark.
-
A structured approach to scalable curation and evaluation.
The proposed pipeline separates inexpensive structural checks from more costly semantic and multimodal assessments. Resumable scoring and structured outputs are useful engineering choices for extending the evaluation to additional models. These are practical strengths of the design, although the reported accounting and protocols need reconciliation.
-
Category-level analyses across multiple models.
The manuscript reports results for 16 models and a human baseline rather than relying on a single model or aggregate score. This evaluation design can expose uneven capability profiles and help distinguish difficult spatial task families. Its potential value should be separated from the numerical and interpretive concerns discussed below.
-
An attempt to validate automated judging with human annotations.
The authors report a 175-example final-answer judging study with three human annotators. Appendix E.2.4 additionally describes a 50-item human check of cluster-to-taxonomy mappings. Including these checks is preferable to relying solely on unvalidated automated judgments, although their endpoints, achieved results, and evidential scope need to be reported more clearly.
Critical weaknesses
-
The curation accounting contains arithmetically impossible transitions.
Table 7 (p. 46) reports 1,943 survivors after R3, followed by 9,812 survivors after R4 removes only four examples. Even if the four Stage-3 criteria were applied simultaneously rather than sequentially, R1 alone removes 4,100 of 12,914 examples, leaving at most 8,814 candidates that could pass all criteria—not 9,812.
The same table reports 24,830 minus 9,088 as 15,792 rather than 15,742, and 9,812 minus 405 as 9,411 rather than 9,407. The raw pool is over 60,000 in the main text but approximately 50,000 in Appendix C.
These contradictions prevent reconstruction of the reported filtering process. Since the curation pipeline is a principal contribution, the discrepancies require reconciliation with the actual dataset manifest and stage-level logs.
-
The detailed task inventory does not match the declared taxonomy or dataset size.
Table 5 (p. 33) explicitly lists 57 named task rows whose counts sum to 6,982, whereas its total row and Table 6 report 65 tasks and 7,838 examples.
Physical Reasoning sums to 1,066 rather than 1,312; Knowledge-Grounded Reasoning to 1,821 rather than 2,332; Quantitative Reasoning to 453 rather than 503; and Perceptual Understanding to 373 rather than 422. Some per-category task counts also differ: Table 5 lists three relational tasks, while Table 6 reports ten.
Omitted rows or different versions could explain this, but neither is identified. The concern is an unreconciled inventory, not a demonstrated absence of examples from the underlying dataset. Nevertheless, readers need an authoritative task list to understand what was actually evaluated.
-
Several central scores and figure–table correspondences cannot be reconciled.
Section 3.1 states that overall accuracy is instance-weighted. Using Table 1's displayed category counts and accuracies gives approximately 45.4081% for Gemini-3.1-Flash-Lite, 23.4660% for Qwen3.5-9B, and 22.2833% for Aya-Vision-32B, versus reported overall values of 45.20%, 23.10%, and 22.89%.
These differences exceed rounding tolerance under the stated aggregation. Missing-response denominators or a different category-level aggregation could matter, but they are not specified.
-
The actual evaluation protocol is not specified consistently.
Appendix F first gives 1,000-token limits for answering and judging, while F.1 specifies 600 tokens for standard models, 20,000 for reasoning models, and 1,200 for the judge.
Appendix L supplies answer choices and requests a one-sentence final answer, while other sections describe exact choice-letter matching or semantic-equivalence judging. These can be different legitimate evaluation modes, but the manuscript does not map them clearly to the reported results. Free-form output with visible choices also does not, by itself, establish an evaluation free from option-elimination shortcuts.
The human baseline is based on three-person majority aggregation rather than an individual-human average; its construction and relation to the stage-wise human scores need to be unambiguous. Appendix I refers to 13 evaluated models despite 16 in the main evaluation, and uses model names that do not consistently match Table 1. The affected subsets and versions should be identified.
-
The advance over existing comprehensive spatial benchmarks is insufficiently established.
SpatialScore [1] already targets unified multimodal spatial evaluation; OmniSpatial [2] offers a cognitively grounded comprehensive taxonomy; Spatial-DISE [3] provides unified spatial assessment; MMSI-Bench [4] includes multi-image reasoning annotations and automated error analysis; and SPAR-Bench [5] covers single-view and multi-view spatial capabilities.
Yet Table 3 omits SpatialScore and OmniSpatial from the central coverage comparison, although OmniSpatial is discussed elsewhere.
The issue is not that another integrated benchmark cannot be valuable. It is that the manuscript does not establish which new capabilities, failure distinctions, or model-ranking insights are uniquely enabled by FrontierSpatial. Counting categories under different taxonomies cannot alone demonstrate broader coverage, and existing error-analysis work must be considered when positioning the diagnostic contribution.In my opinion, their work is already from a year ago, so it doesn't count as work from the same period and needs to be properly cited in the related work section.
Additional reporting issues
- Table 2's gap header indicates Claude Code minus SpatioXolver, while the nonzero entries use the opposite sign.
- The statement that every model scores below 17% on Quantitative reasoning conflicts with Claude's 17.30% in Table 1.
- Section 3.1 says Table 1 also reports category-macro averages, but no separate macro-average column is shown. This is a reporting omission, not evidence that the reported Kendall correlation is wrong.
- Appendix D.5 says scoring has not been validated against humans, whereas F.2 and E.2.4 describe human checks. The scope or version distinction needs explanation.
- Naming and structural remnants include HAL Spatial, HolisticSpatial, a reference to six primary domains despite seven categories, and unresolved cross-references.
Overall assessment
The integrated resource and diagnostic ambition are potentially useful. However, a benchmark paper must establish a coherent dataset inventory, reproducible scoring, and valid measurement interpretations.
The current manuscript has substantial unresolved numerical inconsistencies and insufficiently supported diagnostic and curation claims. My main reservations therefore concern the reliability and validity of the measurement framework, in addition to its incremental contribution over prior benchmarks.
References relevant to the weaknesses
[1] Haoning Wu et al. SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence. CVPR 2026. Initial preprint: SpatialScore: Towards Unified Evaluation for Multimodal Spatial Understanding, arXiv:2505.17012, 2025.
[2] Mengdi Jia et al. OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models. ICLR 2026. Initial preprint: arXiv:2506.03135, 2025.
[3] Xinmiao Huang et al. Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models. arXiv:2510.13394, 2025.
[4] Sihan Yang et al. MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence. arXiv:2505.23764, 2025.
[5] Jiahui Zhang et al. From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D. arXiv:2503.22976, 2025. Introduces SPAR-Bench.
Questions for authors
-
Authoritative data and results.
Which versioned dataset manifest and scoring records reconcile Tables 5–7, Table 1's overall scores, and Figure 4 versus Table 12?
Please identify the actual task list, per-stage retained/rejected counts, valid-response denominators, and aggregation formulas, and distinguish transcription errors from differences in the evaluated data.
-
Incremental holistic coverage.
Which specific capabilities or diagnostic distinctions are added beyond SpatialScore, OmniSpatial, Spatial-DISE, MMSI-Bench, and SPAR-Bench?
What proportion of the released questions satisfies the stated 25% cross-category composition condition, and how does the mapping between comparison dimensions, subcategories, and item-level skill requirements establish that proportion?
-
Executed evaluation and statistical protocols.
Which token limits, model versions, choice-presentation rules, and scoring modes generated each reported result? How were missing responses and different model subsets handled?
Please also clarify how per-error-type accuracy denominators are defined for successful responses, and distinguish McNemar-based accuracy comparisons from claims about complementary error locations.
-
Validation of stage-wise diagnosis.
Did the Figure 4 judge receive images and model-generated reasoning traces, and what happened when those traces were absent? Please locate the achieved result of the 50-item mapping check.
A targeted human audit of stage-wise labels, using the actual evidence available to the judge, would help distinguish defensible output annotations from inferred or unobservable failure mechanisms.
-
Validation of curation decisions.
How do the knowledge and perception categories survive R3 and R2, and how are distinct questions or ordered image sequences protected during deduplication? Please locate the claimed staged-ablation results.
A stratified audit of retained and rejected examples would help establish whether the pipeline removes invalid or redundant items without disproportionately deleting valid task diversity.
7 comments
Response to Reviewer JA5E [Part 2/6]
W3. Overall scores vs. instance-weighted category cells; Figure 4 vs. Table 12
Definition. Overall is the harness overall_score: the mean of each model's 7,838 per-instance accuracy scores; every instance is scored (failed calls re-submitted, Appendix D.2). The reviewer re-weights the seven displayed category cells. We reproduced the reviewer's values (45.4081, 23.4660, 22.2833) and compared the two aggregations against sampling uncertainty.
Table R3. Overall accuracy under three aggregations (%, n = 7,838; CI = 95% half-width).
| Model | overall_score | Re-weighted cells | Diff. | CI | Macro-average |
|---|---|---|---|---|---|
| Gemini-3.1-Flash-Lite | 45.20 | 45.41 | +0.21 | 1.10 | 43.92 |
| GPT-5-mini | 41.95 | 42.10 | +0.15 | 1.09 | 42.77 |
| o4-mini | 40.32 | 40.40 | +0.08 | 1.09 | 41.91 |
| Claude 4.5 Haiku | 37.45 | 37.45 | 0.00 | 1.07 | 38.82 |
| Gemini-2.5-Flash | 36.99 | 37.20 | +0.21 | 1.07 | 37.18 |
| GLM-4.5V | 35.29 | 35.41 | +0.12 | 1.06 | 37.23 |
| Qwen3.6-35B-A3B | 34.84 | 35.07 | +0.23 | 1.05 | 33.84 |
| Gemma-4-31B-it | 33.50 | 33.59 | +0.09 | 1.04 | 37.16 |
| Qwen3-VL-8B-Instruct | 33.45 | 33.63 | +0.18 | 1.04 | 35.16 |
| Kimi K2.5 | 32.45 | 32.57 | +0.12 | 1.04 | 32.55 |
| Mistral Small 2603 | 27.32 | 27.44 | +0.12 | 0.99 | 30.63 |
| GPT-4o-mini | 25.42 | 25.55 | +0.13 | 0.96 | 27.82 |
| Qwen3.5-9B | 23.10 | 23.47 | +0.37 | 0.93 | 25.75 |
| Aya-Vision-32B | 22.89 | 22.28 | -0.61 | 0.93 | 25.83 |
| Llama-3.2-11B-Vision-Ins. | 22.54 | 22.67 | +0.13 | 0.93 | 24.30 |
| Gemma-3-4B-it | 19.66 | 19.68 | +0.02 | 0.88 | 20.76 |
| Human (majority) | 91.22 | 91.22 | 0.00 | n/a | 91.91 |
Mean absolute difference is 0.17 points; the maximum (0.61, Aya-Vision-32B) is 0.66 of its CI half-width, and 15 of 16 differences are at most 0.40 of it. Re-weighting changes the ranking only by two adjacent swaps between statistical ties (Gemma-4-31B-it vs. Qwen3-VL-8B, 0.05 apart; Aya-Vision-32B vs. Llama-3.2-11B, 0.35 apart), Kendall tau = 0.967. The macro-average (now a Table 1 column) gives tau = 0.933, a maximum shift of two positions, and unchanged top four and bottom two, confirming Section 3.1. The best model, the 46.0-point human-model gap, and all category bottlenecks hold under all three aggregations. The Table 1 caption now states the formula.
Figure 4 vs. Table 12. All 336 model cells of Figure 4 match Tables 11 to 13 after rounding (9 differ by 0.05 from rounding ties). The only difference is Table 12's human row, a leftover from an earlier template (Tables 11 and 13 have none). Figure 4 and Section 4.2 (100.0, 99.9, 99.7) are authoritative; the row is removed.
W4. Executed protocol
- Token limits. All runs used Appendix F.1: 600 tokens (standard), 20,000 (Claude 4.5 Haiku, GPT-4o-mini, GPT-5-mini, o4-mini, Kimi K2.5, GLM-4.5V), judge 1,200. The budget is a ceiling, preventing truncation without affecting models that stop earlier. The 1,000 in Appendices D and F is the harness default and is removed.
- Choices and scoring mode. One mode for all reported results: choices shown as context (Appendix L), free-form answers, judge semantic equivalence. Exact letter matching (Appendices D.2, E.1) is an unused harness option. Section 4 and the Table 1 caption state this.
- Option elimination. R5 gave a multimodal model every question with its choices but no image and removed the 405 items it answered correctly with high confidence, the established control for option and text shortcuts (MMStar [8]; MMMU-Pro [9]). For many items the choices are bare labels with content only in the image ("A, B, C, D" on folding, 3D assembly, and cube-net items, Appendix M), where textual elimination is impossible.
- Human baseline. Three annotators (high school; STEM undergraduate; non-STEM undergraduate) answered all 7,838 items in open form, about 150 hours each over one month (23,514 responses), scored by the same judge as the models. An item is correct when at least two of three are correct; the same rule applies to each stage dimension in Figure 4. Consensus is deliberate: over 150 hours, individual responses include attention and fatigue lapses unrelated to spatial competence, and consensus scoring is standard for estimating attainable human performance (VQA [10]). Full-benchmark coverage exceeds common practice; VSI-Bench measured humans on a 400-question subset [11].
- Appendix I. Its qualitative "affected models" lists come from the 13 models evaluated when the taxonomy was written; Qwen3.5-9B, Aya-Vision-32B, and Qwen3.6-35B-A3B were added later. The E1 to E7 schema is model-agnostic and fixed before evaluation (Appendix E, Step 2d). API identifiers (e.g., z_ai_glm_4.5v, google_gemini_3.1_flash_lite_preview) are replaced by Table 1 names.
Response to Reviewer JA5E [Part 3/6]
W5. Advance over SpatialScore, OmniSpatial, Spatial-DISE, MMSI-Bench, SPAR-Bench
We agree that all five must be cited and compared, and that cross-taxonomy dimension counts are insufficient. SpatialScore and SPAR-Bench are now in the related work, and SpatialScore, OmniSpatial, and SPAR-Bench are rows of Table 3. Each differentiator below is measured directly rather than by aligning taxonomies.
Table R4. What each benchmark establishes and what FrontierSpatial adds (values from each paper).
| SpatialScore [1] | OmniSpatial [2] | Spatial-DISE [3] | MMSI-Bench [4] | SPAR-Bench [5] | FrontierSpatial | |
|---|---|---|---|---|---|---|
| Items | 28K (Hard: 1,400) | about 1.5K | 559 | 1,000 | 7,207 | 7,838 (11,825 images) |
| Origin | VGBench (3D templates) + 11 datasets | Manual, internet images | Real + Blender synthetic | Expert-written | Generated from 3D indoor scenes | Curated from 22 datasets |
| Taxonomy | 3D spatial understanding | 4 dimensions, 50 subcategories | 2 x 2 quadrants, 10 tasks | 10 tasks + multi-step | 20 tasks, 3 levels | 7 categories, 19 sub-categories, 65 tasks |
| Knowledge-grounded category | No | No | No | No | No | Yes (2,332 items, 18 tasks) |
| Format | Mixed | Multiple choice | Multiple choice (random 24.8%) | Multiple choice | Numerical + classification | Free-form, judge-scored (kappa = 1.000 vs. humans, n = 175) |
- Containment. MMSI-Bench (623 items) and Spatial-DISE (301) are curated components of FrontierSpatial (Table 4), evaluated jointly with 6,914 items from 20 other sources under one protocol. The relation is inclusion, not competition.
- Ranking instability only joint evaluation can reveal. Across the 16 models, the Kendall tau between category rankings averages 0.54 (0.36 to 0.68 over 21 pairs), and 6 of 16 models shift 5 or more positions between overall and some category rank: Gemma-4-31B-it is 2nd in Physical and Perceptual but 11th in Knowledge (8th overall); Claude 4.5 Haiku is 1st in Quantitative but 10th in Positional & Geometric (4th overall); Mistral Small 2603 is 5th in Temporal and 16th in Perceptual (11th overall). A single-family benchmark (3D geometry, as in SpatialScore and SPAR-Bench, or cognitive transformations, as in Spatial-DISE) would produce rankings the full profile contradicts for these models.
- Rationale-free diagnostics. MMSI-Bench's error analysis relies on human-written reasoning for each of its 1,000 items, which most of our 22 sources lack. Our pipeline clusters failure patterns (question, gold, prediction), maps them to a schema fixed before evaluation, applies paired statistics, and assigns new models by nearest centroid, so one diagnostic layer spans all seven families. Its stage annotations are validated against humans (Table R6).
- Disjoint consolidation. Of the nine SpatialScore sources named in its paper (MMVP, RealWorldQA, SpatialSense, SpatialBench, Q-Spatial-Bench, CV-Bench, VSR, 3DSRBench, VSI-Bench), only Q-Spatial-Bench (82 of our items) is among ours. FrontierSpatial adds knowledge-grounded (2,332), physical and driving (1,312), relational and causal (455), and perceptual (422) families, with cross-source deduplication (36.4% of Stage 1 survivors) and blind no-image filtering.
- Full-benchmark human baseline on all 7,838 items (W4). Integrating existing resources under one harness with a diagnostic layer is an established contribution type (HELM [6]; HAL, ICLR 2026 [7]; SpatialScore itself integrates 11 datasets [1]); its value rests on coverage, curation quality, and the insights enabled, which points 1 to 5 quantify.
Revised text (Appendix A, summarized in Section 1): "SpatialScore integrates VGBench with 11 datasets (28K samples) focused on 3D perception; OmniSpatial provides about 1.5K manual questions across 4 dimensions; Spatial-DISE organizes 559 items into four cognitive quadrants; MMSI-Bench contributes 1,000 expert multi-image questions with annotated reasoning; SPAR-Bench evaluates 20 tasks from 3D indoor scenes. FrontierSpatial complements these and includes curated subsets of MMSI-Bench (623) and Spatial-DISE (301). It contributes (i) one protocol over 7 categories and 65 tasks, including knowledge-grounded reasoning (2,332 items), with each category estimated within +/- 6.64 points at 95% confidence and a full-benchmark human baseline; (ii) consolidation-specific quality control (36.4% cross-source near-duplicates removed; blind no-image filtering); and (iii) rationale-free error analysis applicable across heterogeneous sources. Joint evaluation reveals ranking instability across families (mean Kendall tau = 0.54)." "First holistic framework" is replaced by these claims.
Response to Reviewer JA5E [Part 1/6]
We thank the reviewer for an unusually thorough check of sums, aggregations, and transitions, and for recognizing the value of a unified interface across seven spatial families, the staged curation design, and category-level profiles for 16 models plus a human baseline. We agree that "an integrated measurement resource can be valuable even without being the first comprehensive benchmark." In brief: (i) each numerical point traces to a specific reporting step (8 rows omitted from Table 5, a mislabeled Stage 3 layout in Table 7, a leftover human row in Table 12), and the corrected versions close arithmetically; (ii) no ranking or conclusion changes under any aggregation; (iii) a new human audit validates the stage labels (kappa = 0.84 to 0.91); (iv) FrontierSpatial's distinct contribution is measured directly against the five benchmarks. We have also highlighted all changes in the revised manuscript in blue.
W1. Table 7 transitions; 60K+ vs. 50K raw pool
Stage 3 scores all four criteria in one LLM call and removes an instance failing any of them (Appendix C.3: "must pass all four thresholds simultaneously"). There is no sequential pool, so each criterion should report instances flagged, plus the union. The submitted table placed these in a "Remaining" layout, producing the transitions the reviewer identified. Reconciled from the Stage 3 logs:
Revised Table 7.
Every row closes: each "Reduced by" equals the difference of consecutive "Remaining" values (2a+2b corrected from a one-digit typo to 9,038), reductions sum to 42,162 = 50,000 minus 7,838, and each flag count is below the 3,098 union. The flags converge: 7,918 flags on 3,098 items (2.56 per item) imply at least 1,607 removed items (52%) failed two or more criteria. Appendix C.3/C.4 headings and per-criterion numbers are updated.
Raw pool and dataset count. "60K+" counts all instances in the selected source subsets combined; Table 7 starts from the 50,000 admitted into the unified image-question-answer schema, which is also the base of "Stage 1 removes about 50%" (50.3%). Both are now defined in Section 3.1. Table 4 lists 22 distinct datasets whose counts sum exactly to 7,838; the "23" counted raw-pool source identifiers, where MMSI-Bench appears under more than one identifier (Appendix C.1d). The text now reads 22 throughout.
W2. Task inventory (Table 5)
Appendix B already gives counts for all 65 tasks, and they reconcile exactly with Table 1 and 7,838. Table 5 omitted 8 rows:
Table R1. Rows omitted from Table 5 (counts from Appendix B).
Table R2. Reconciled inventory (tasks / items).
Table 6's per-category columns were shifted between rows (totals 19 and 65 correct); they now read 3/9, 5/20, 3/18, 2/3, 2/4, 2/5, 2/6. Relational has 3 tasks. Two Appendix B subtotal typos are fixed (Positional Relations = 390; Positional & Geometric = 2,596). Each item carries its task label in the manifest.