acceptodds
Under review as a conference paper at ICLR 2027

FrontierSpatial: Holistic Evaluation of Spatial Reasoning in Large Multimodal Models

12 views

Abstract

Spatial reasoning is foundational to intelligent systems, yet its evaluation in multimodal large language models (MLLMs) remains fragmented across narrow benchmarks that test isolated skills. We introduce ***FrontierSpatial***, a unified benchmark and evaluation framework for large-scale spatial reasoning. We consolidate 23 existing datasets into **7,838 questions paired with 11,825 images**, organized into 7 categories, 19 sub-categories, and 65 fine-grained tasks spanning physical, geometric, knowledge-grounded, relational/causal, temporal, quantitative, and perceptual reasoning. Beyond the benchmark, we provide (1) an automated curation pipeline that distills over 60,000 raw instances into a balanced dataset and generalizes to other domains, (2) an automated evaluation harness for reproducible assessment of 16 state-of-the-art MLLMs, and (3) an automated error-analysis harness for fine-grained diagnosis across the task hierarchy. Our evaluation reveals large and uneven performance gaps across spatial skills, exposing systematic weaknesses that remain hidden in isolated benchmarks. ***FrontierSpatial*** provides a rigorous foundation for systematic, fine-grained evaluation and deeper understanding of multimodal spatial reasoning.

Primary area:
datasets and benchmarks
Keywords:
spatial reasoning, vision-language models, spatial cognition, evaluation framework, evaluation harness, error analysis, diagnostic benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JA5EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FrontierSpatial, a benchmark and evaluation framework for visual spatial reasoning in multimodal language models. The manuscript reports 7,838 questions paired with 11,825 images, drawn from 23 existing datasets and organized into seven categories, 19 subcategories, and 65 fine-grained tasks. Its four-stage curation pipeline combines structural filtering, visual deduplication, text-based quality assessment, and multimodal validation.

The authors also propose an automated evaluation and error-analysis pipeline. It scores final-answer correctness and several spatial reasoning dimensions, clusters failed responses, and maps the resulting clusters to a hierarchical error taxonomy. The reported evaluation covers 16 models and a human baseline, emphasizing a gap between identifying spatial information and composing it into correct reasoning. Existing agentic systems are additionally evaluated on a small hard-case subset. The intended contribution is an integrated measurement and diagnostic framework, rather than a new spatial-reasoning model or agent architecture.

Critical strengths
  1. A practically useful attempt to unify heterogeneous evaluation resources.

    Bringing physical, geometric, relational, temporal, quantitative, perceptual, and knowledge-grounded tasks into a common interface could reduce the effort required to compare spatial capabilities across datasets. An integrated measurement resource can be valuable even without being the first comprehensive benchmark.

  2. A structured approach to scalable curation and evaluation.

    The proposed pipeline separates inexpensive structural checks from more costly semantic and multimodal assessments. Resumable scoring and structured outputs are useful engineering choices for extending the evaluation to additional models. These are practical strengths of the design, although the reported accounting and protocols need reconciliation.

  3. Category-level analyses across multiple models.

    The manuscript reports results for 16 models and a human baseline rather than relying on a single model or aggregate score. This evaluation design can expose uneven capability profiles and help distinguish difficult spatial task families. Its potential value should be separated from the numerical and interpretive concerns discussed below.

  4. An attempt to validate automated judging with human annotations.

    The authors report a 175-example final-answer judging study with three human annotators. Appendix E.2.4 additionally describes a 50-item human check of cluster-to-taxonomy mappings. Including these checks is preferable to relying solely on unvalidated automated judgments, although their endpoints, achieved results, and evidential scope need to be reported more clearly.

Critical weaknesses
  1. The curation accounting contains arithmetically impossible transitions.

    Table 7 (p. 46) reports 1,943 survivors after R3, followed by 9,812 survivors after R4 removes only four examples. Even if the four Stage-3 criteria were applied simultaneously rather than sequentially, R1 alone removes 4,100 of 12,914 examples, leaving at most 8,814 candidates that could pass all criteria—not 9,812.

    The same table reports 24,830 minus 9,088 as 15,792 rather than 15,742, and 9,812 minus 405 as 9,411 rather than 9,407. The raw pool is over 60,000 in the main text but approximately 50,000 in Appendix C.

    These contradictions prevent reconstruction of the reported filtering process. Since the curation pipeline is a principal contribution, the discrepancies require reconciliation with the actual dataset manifest and stage-level logs.

  2. The detailed task inventory does not match the declared taxonomy or dataset size.

    Table 5 (p. 33) explicitly lists 57 named task rows whose counts sum to 6,982, whereas its total row and Table 6 report 65 tasks and 7,838 examples.

    Physical Reasoning sums to 1,066 rather than 1,312; Knowledge-Grounded Reasoning to 1,821 rather than 2,332; Quantitative Reasoning to 453 rather than 503; and Perceptual Understanding to 373 rather than 422. Some per-category task counts also differ: Table 5 lists three relational tasks, while Table 6 reports ten.

    Omitted rows or different versions could explain this, but neither is identified. The concern is an unreconciled inventory, not a demonstrated absence of examples from the underlying dataset. Nevertheless, readers need an authoritative task list to understand what was actually evaluated.

  3. Several central scores and figure–table correspondences cannot be reconciled.

    Section 3.1 states that overall accuracy is instance-weighted. Using Table 1's displayed category counts and accuracies gives approximately 45.4081% for Gemini-3.1-Flash-Lite, 23.4660% for Qwen3.5-9B, and 22.2833% for Aya-Vision-32B, versus reported overall values of 45.20%, 23.10%, and 22.89%.

    These differences exceed rounding tolerance under the stated aggregation. Missing-response denominators or a different category-level aggregation could matter, but they are not specified.

  4. The actual evaluation protocol is not specified consistently.

    Appendix F first gives 1,000-token limits for answering and judging, while F.1 specifies 600 tokens for standard models, 20,000 for reasoning models, and 1,200 for the judge.

    Appendix L supplies answer choices and requests a one-sentence final answer, while other sections describe exact choice-letter matching or semantic-equivalence judging. These can be different legitimate evaluation modes, but the manuscript does not map them clearly to the reported results. Free-form output with visible choices also does not, by itself, establish an evaluation free from option-elimination shortcuts.

    The human baseline is based on three-person majority aggregation rather than an individual-human average; its construction and relation to the stage-wise human scores need to be unambiguous. Appendix I refers to 13 evaluated models despite 16 in the main evaluation, and uses model names that do not consistently match Table 1. The affected subsets and versions should be identified.

  5. The advance over existing comprehensive spatial benchmarks is insufficiently established.

    SpatialScore [1] already targets unified multimodal spatial evaluation; OmniSpatial [2] offers a cognitively grounded comprehensive taxonomy; Spatial-DISE [3] provides unified spatial assessment; MMSI-Bench [4] includes multi-image reasoning annotations and automated error analysis; and SPAR-Bench [5] covers single-view and multi-view spatial capabilities.

    Yet Table 3 omits SpatialScore and OmniSpatial from the central coverage comparison, although OmniSpatial is discussed elsewhere.

    The issue is not that another integrated benchmark cannot be valuable. It is that the manuscript does not establish which new capabilities, failure distinctions, or model-ranking insights are uniquely enabled by FrontierSpatial. Counting categories under different taxonomies cannot alone demonstrate broader coverage, and existing error-analysis work must be considered when positioning the diagnostic contribution.In my opinion, their work is already from a year ago, so it doesn't count as work from the same period and needs to be properly cited in the related work section.

Additional reporting issues

  • Table 2's gap header indicates Claude Code minus SpatioXolver, while the nonzero entries use the opposite sign.
  • The statement that every model scores below 17% on Quantitative reasoning conflicts with Claude's 17.30% in Table 1.
  • Section 3.1 says Table 1 also reports category-macro averages, but no separate macro-average column is shown. This is a reporting omission, not evidence that the reported Kendall correlation is wrong.
  • Appendix D.5 says scoring has not been validated against humans, whereas F.2 and E.2.4 describe human checks. The scope or version distinction needs explanation.
  • Naming and structural remnants include HAL Spatial, HolisticSpatial, a reference to six primary domains despite seven categories, and unresolved cross-references.

Overall assessment

The integrated resource and diagnostic ambition are potentially useful. However, a benchmark paper must establish a coherent dataset inventory, reproducible scoring, and valid measurement interpretations.

The current manuscript has substantial unresolved numerical inconsistencies and insufficiently supported diagnostic and curation claims. My main reservations therefore concern the reliability and validity of the measurement framework, in addition to its incremental contribution over prior benchmarks.

References relevant to the weaknesses

[1] Haoning Wu et al. SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence. CVPR 2026. Initial preprint: SpatialScore: Towards Unified Evaluation for Multimodal Spatial Understanding, arXiv:2505.17012, 2025.

[2] Mengdi Jia et al. OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models. ICLR 2026. Initial preprint: arXiv:2506.03135, 2025.

[3] Xinmiao Huang et al. Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models. arXiv:2510.13394, 2025.

[4] Sihan Yang et al. MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence. arXiv:2505.23764, 2025.

[5] Jiahui Zhang et al. From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D. arXiv:2503.22976, 2025. Introduces SPAR-Bench.

Questions for authors
  1. Authoritative data and results.

    Which versioned dataset manifest and scoring records reconcile Tables 5–7, Table 1's overall scores, and Figure 4 versus Table 12?

    Please identify the actual task list, per-stage retained/rejected counts, valid-response denominators, and aggregation formulas, and distinguish transcription errors from differences in the evaluated data.

  2. Incremental holistic coverage.

    Which specific capabilities or diagnostic distinctions are added beyond SpatialScore, OmniSpatial, Spatial-DISE, MMSI-Bench, and SPAR-Bench?

    What proportion of the released questions satisfies the stated 25% cross-category composition condition, and how does the mapping between comparison dimensions, subcategories, and item-level skill requirements establish that proportion?

  3. Executed evaluation and statistical protocols.

    Which token limits, model versions, choice-presentation rules, and scoring modes generated each reported result? How were missing responses and different model subsets handled?

    Please also clarify how per-error-type accuracy denominators are defined for successful responses, and distinguish McNemar-based accuracy comparisons from claims about complementary error locations.

  4. Validation of stage-wise diagnosis.

    Did the Figure 4 judge receive images and model-generated reasoning traces, and what happened when those traces were absent? Please locate the achieved result of the 50-item mapping check.

    A targeted human audit of stage-wise labels, using the actual evidence available to the judge, would help distinguish defensible output annotations from inferred or unobservable failure mechanisms.

  5. Validation of curation decisions.

    How do the knowledge and perception categories survive R3 and R2, and how are distinct questions or ordered image sequences protected during deduplication? Please locate the claimed staged-ablation results.

    A stratified audit of retained and rejected examples would help establish whether the pipeline removes invalid or redundant items without disproportionately deleting valid task diversity.

7 comments
AuthorsOpenReview ↗

Response to Reviewer JA5E [Part 1/6]

We thank the reviewer for an unusually thorough check of sums, aggregations, and transitions, and for recognizing the value of a unified interface across seven spatial families, the staged curation design, and category-level profiles for 16 models plus a human baseline. We agree that "an integrated measurement resource can be valuable even without being the first comprehensive benchmark." In brief: (i) each numerical point traces to a specific reporting step (8 rows omitted from Table 5, a mislabeled Stage 3 layout in Table 7, a leftover human row in Table 12), and the corrected versions close arithmetically; (ii) no ranking or conclusion changes under any aggregation; (iii) a new human audit validates the stage labels (kappa = 0.84 to 0.91); (iv) FrontierSpatial's distinct contribution is measured directly against the five benchmarks. We have also highlighted all changes in the revised manuscript in blue.


W1. Table 7 transitions; 60K+ vs. 50K raw pool

Stage 3 scores all four criteria in one LLM call and removes an instance failing any of them (Appendix C.3: "must pass all four thresholds simultaneously"). There is no sequential pool, so each criterion should report instances flagged, plus the union. The submitted table placed these in a "Remaining" layout, producing the transitions the reviewer identified. Reconciled from the Stage 3 logs:

Revised Table 7.

StageStepFlaggedRemainingReduced by
Raw pool50,000
Stage 11a / 1b / 1c / 1d / 1e44,402 / 43,852 / 35,232 / 26,683 / 24,8305,598 / 550 / 8,620 / 8,549 / 1,853
Stage 22a+2b near-duplicates15,7929,038
2c domain clustering12,9142,878
Stage 3R1 / R2 / R3 / R42,772 / 2,432 / 2,710 / 4
R1 to R4 union3,0989,8163,098
Stage 4R5 / R6 / R79,411 / 8,912 / 7,838405 / 499 / 1,074
Final7,83842,162

Every row closes: each "Reduced by" equals the difference of consecutive "Remaining" values (2a+2b corrected from a one-digit typo to 9,038), reductions sum to 42,162 = 50,000 minus 7,838, and each flag count is below the 3,098 union. The flags converge: 7,918 flags on 3,098 items (2.56 per item) imply at least 1,607 removed items (52%) failed two or more criteria. Appendix C.3/C.4 headings and per-criterion numbers are updated.

Raw pool and dataset count. "60K+" counts all instances in the selected source subsets combined; Table 7 starts from the 50,000 admitted into the unified image-question-answer schema, which is also the base of "Stage 1 removes about 50%" (50.3%). Both are now defined in Section 3.1. Table 4 lists 22 distinct datasets whose counts sum exactly to 7,838; the "23" counted raw-pool source identifiers, where MMSI-Bench appears under more than one identifier (Appendix C.1d). The text now reads 22 throughout.

W2. Task inventory (Table 5)

Appendix B already gives counts for all 65 tasks, and they reconcile exactly with Table 1 and 7,838. Table 5 omitted 8 rows:

Table R1. Rows omitted from Table 5 (counts from Appendix B).

CategoryTasknLocation
PhysicalProperty Inference; Collision74; 172B.1.3
Knowledge-GroundedGeometric Reasoning & Visual Problem Solving; Geometric Construction & Optimization; Circle Geometry234; 95; 132B.3.1
Knowledge-GroundedScientific Reasoning50B.3.3
QuantitativeRelational Counting50B.6.2
PerceptualAppearance49B.7.1
Total8 tasks856

Table R2. Reconciled inventory (tasks / items).

CategoryTable 5 submittedOmittedReconciledTable 1 headerSub-categories
Physical7 / 1,0662 (246)9 / 1,3121,3123
Positional & Geometric20 / 2,596020 / 2,5962,5965
Knowledge-Grounded14 / 1,8214 (511)18 / 2,3322,3323
Relational, Logical & Causal3 / 45503 / 4554552
Temporal & Dynamic4 / 21804 / 2182182
Quantitative & Comparative4 / 4531 (50)5 / 5035032
Perceptual5 / 3731 (49)6 / 4224222
Total57 / 6,9828 (856)65 / 7,8387,83819

Table 6's per-category columns were shifted between rows (totals 19 and 65 correct); they now read 3/9, 5/20, 3/18, 2/3, 2/4, 2/5, 2/6. Relational has 3 tasks. Two Appendix B subtotal typos are fixed (Positional Relations = 390; Positional & Geometric = 2,596). Each item carries its task label in the manifest.

AuthorsOpenReview ↗

Response to Reviewer JA5E [Part 2/6]

W3. Overall scores vs. instance-weighted category cells; Figure 4 vs. Table 12

Definition. Overall is the harness overall_score: the mean of each model's 7,838 per-instance accuracy scores; every instance is scored (failed calls re-submitted, Appendix D.2). The reviewer re-weights the seven displayed category cells. We reproduced the reviewer's values (45.4081, 23.4660, 22.2833) and compared the two aggregations against sampling uncertainty.

Table R3. Overall accuracy under three aggregations (%, n = 7,838; CI = 95% half-width).

Modeloverall_scoreRe-weighted cellsDiff.CIMacro-average
Gemini-3.1-Flash-Lite45.2045.41+0.211.1043.92
GPT-5-mini41.9542.10+0.151.0942.77
o4-mini40.3240.40+0.081.0941.91
Claude 4.5 Haiku37.4537.450.001.0738.82
Gemini-2.5-Flash36.9937.20+0.211.0737.18
GLM-4.5V35.2935.41+0.121.0637.23
Qwen3.6-35B-A3B34.8435.07+0.231.0533.84
Gemma-4-31B-it33.5033.59+0.091.0437.16
Qwen3-VL-8B-Instruct33.4533.63+0.181.0435.16
Kimi K2.532.4532.57+0.121.0432.55
Mistral Small 260327.3227.44+0.120.9930.63
GPT-4o-mini25.4225.55+0.130.9627.82
Qwen3.5-9B23.1023.47+0.370.9325.75
Aya-Vision-32B22.8922.28-0.610.9325.83
Llama-3.2-11B-Vision-Ins.22.5422.67+0.130.9324.30
Gemma-3-4B-it19.6619.68+0.020.8820.76
Human (majority)91.2291.220.00n/a91.91

Mean absolute difference is 0.17 points; the maximum (0.61, Aya-Vision-32B) is 0.66 of its CI half-width, and 15 of 16 differences are at most 0.40 of it. Re-weighting changes the ranking only by two adjacent swaps between statistical ties (Gemma-4-31B-it vs. Qwen3-VL-8B, 0.05 apart; Aya-Vision-32B vs. Llama-3.2-11B, 0.35 apart), Kendall tau = 0.967. The macro-average (now a Table 1 column) gives tau = 0.933, a maximum shift of two positions, and unchanged top four and bottom two, confirming Section 3.1. The best model, the 46.0-point human-model gap, and all category bottlenecks hold under all three aggregations. The Table 1 caption now states the formula.

Figure 4 vs. Table 12. All 336 model cells of Figure 4 match Tables 11 to 13 after rounding (9 differ by 0.05 from rounding ties). The only difference is Table 12's human row, a leftover from an earlier template (Tables 11 and 13 have none). Figure 4 and Section 4.2 (100.0, 99.9, 99.7) are authoritative; the row is removed.

W4. Executed protocol

  • Token limits. All runs used Appendix F.1: 600 tokens (standard), 20,000 (Claude 4.5 Haiku, GPT-4o-mini, GPT-5-mini, o4-mini, Kimi K2.5, GLM-4.5V), judge 1,200. The budget is a ceiling, preventing truncation without affecting models that stop earlier. The 1,000 in Appendices D and F is the harness default and is removed.
  • Choices and scoring mode. One mode for all reported results: choices shown as context (Appendix L), free-form answers, judge semantic equivalence. Exact letter matching (Appendices D.2, E.1) is an unused harness option. Section 4 and the Table 1 caption state this.
  • Option elimination. R5 gave a multimodal model every question with its choices but no image and removed the 405 items it answered correctly with high confidence, the established control for option and text shortcuts (MMStar [8]; MMMU-Pro [9]). For many items the choices are bare labels with content only in the image ("A, B, C, D" on folding, 3D assembly, and cube-net items, Appendix M), where textual elimination is impossible.
  • Human baseline. Three annotators (high school; STEM undergraduate; non-STEM undergraduate) answered all 7,838 items in open form, about 150 hours each over one month (23,514 responses), scored by the same judge as the models. An item is correct when at least two of three are correct; the same rule applies to each stage dimension in Figure 4. Consensus is deliberate: over 150 hours, individual responses include attention and fatigue lapses unrelated to spatial competence, and consensus scoring is standard for estimating attainable human performance (VQA [10]). Full-benchmark coverage exceeds common practice; VSI-Bench measured humans on a 400-question subset [11].
  • Appendix I. Its qualitative "affected models" lists come from the 13 models evaluated when the taxonomy was written; Qwen3.5-9B, Aya-Vision-32B, and Qwen3.6-35B-A3B were added later. The E1 to E7 schema is model-agnostic and fixed before evaluation (Appendix E, Step 2d). API identifiers (e.g., z_ai_glm_4.5v, google_gemini_3.1_flash_lite_preview) are replaced by Table 1 names.
AuthorsOpenReview ↗

Response to Reviewer JA5E [Part 3/6]

W5. Advance over SpatialScore, OmniSpatial, Spatial-DISE, MMSI-Bench, SPAR-Bench

We agree that all five must be cited and compared, and that cross-taxonomy dimension counts are insufficient. SpatialScore and SPAR-Bench are now in the related work, and SpatialScore, OmniSpatial, and SPAR-Bench are rows of Table 3. Each differentiator below is measured directly rather than by aligning taxonomies.

Table R4. What each benchmark establishes and what FrontierSpatial adds (values from each paper).

SpatialScore [1]OmniSpatial [2]Spatial-DISE [3]MMSI-Bench [4]SPAR-Bench [5]FrontierSpatial
Items28K (Hard: 1,400)about 1.5K5591,0007,2077,838 (11,825 images)
OriginVGBench (3D templates) + 11 datasetsManual, internet imagesReal + Blender syntheticExpert-writtenGenerated from 3D indoor scenesCurated from 22 datasets
Taxonomy3D spatial understanding4 dimensions, 50 subcategories2 x 2 quadrants, 10 tasks10 tasks + multi-step20 tasks, 3 levels7 categories, 19 sub-categories, 65 tasks
Knowledge-grounded categoryNoNoNoNoNoYes (2,332 items, 18 tasks)
FormatMixedMultiple choiceMultiple choice (random 24.8%)Multiple choiceNumerical + classificationFree-form, judge-scored (kappa = 1.000 vs. humans, n = 175)
  1. Containment. MMSI-Bench (623 items) and Spatial-DISE (301) are curated components of FrontierSpatial (Table 4), evaluated jointly with 6,914 items from 20 other sources under one protocol. The relation is inclusion, not competition.
  2. Ranking instability only joint evaluation can reveal. Across the 16 models, the Kendall tau between category rankings averages 0.54 (0.36 to 0.68 over 21 pairs), and 6 of 16 models shift 5 or more positions between overall and some category rank: Gemma-4-31B-it is 2nd in Physical and Perceptual but 11th in Knowledge (8th overall); Claude 4.5 Haiku is 1st in Quantitative but 10th in Positional & Geometric (4th overall); Mistral Small 2603 is 5th in Temporal and 16th in Perceptual (11th overall). A single-family benchmark (3D geometry, as in SpatialScore and SPAR-Bench, or cognitive transformations, as in Spatial-DISE) would produce rankings the full profile contradicts for these models.
  3. Rationale-free diagnostics. MMSI-Bench's error analysis relies on human-written reasoning for each of its 1,000 items, which most of our 22 sources lack. Our pipeline clusters failure patterns (question, gold, prediction), maps them to a schema fixed before evaluation, applies paired statistics, and assigns new models by nearest centroid, so one diagnostic layer spans all seven families. Its stage annotations are validated against humans (Table R6).
  4. Disjoint consolidation. Of the nine SpatialScore sources named in its paper (MMVP, RealWorldQA, SpatialSense, SpatialBench, Q-Spatial-Bench, CV-Bench, VSR, 3DSRBench, VSI-Bench), only Q-Spatial-Bench (82 of our items) is among ours. FrontierSpatial adds knowledge-grounded (2,332), physical and driving (1,312), relational and causal (455), and perceptual (422) families, with cross-source deduplication (36.4% of Stage 1 survivors) and blind no-image filtering.
  5. Full-benchmark human baseline on all 7,838 items (W4). Integrating existing resources under one harness with a diagnostic layer is an established contribution type (HELM [6]; HAL, ICLR 2026 [7]; SpatialScore itself integrates 11 datasets [1]); its value rests on coverage, curation quality, and the insights enabled, which points 1 to 5 quantify.

Revised text (Appendix A, summarized in Section 1): "SpatialScore integrates VGBench with 11 datasets (28K samples) focused on 3D perception; OmniSpatial provides about 1.5K manual questions across 4 dimensions; Spatial-DISE organizes 559 items into four cognitive quadrants; MMSI-Bench contributes 1,000 expert multi-image questions with annotated reasoning; SPAR-Bench evaluates 20 tasks from 3D indoor scenes. FrontierSpatial complements these and includes curated subsets of MMSI-Bench (623) and Spatial-DISE (301). It contributes (i) one protocol over 7 categories and 65 tasks, including knowledge-grounded reasoning (2,332 items), with each category estimated within +/- 6.64 points at 95% confidence and a full-benchmark human baseline; (ii) consolidation-specific quality control (36.4% cross-source near-duplicates removed; blind no-image filtering); and (iii) rationale-free error analysis applicable across heterogeneous sources. Joint evaluation reveals ranking instability across families (mean Kendall tau = 0.54)." "First holistic framework" is replaced by these claims.

Reviewer_fdcgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

FrontierSpatial presents 7838 questions and 11825 images from a reported 23 datasets, organized into seven categories and 65 spatial-reasoning tasks. It includes a unified taxonomy with automated curation, evaluation, and error analysis. Validation includes 16 models, a human baseline, a 175-sample judge-agreement study, and agent experiments on 175 hard cases.

Critical strengths

The category breakdowns reveal weaknesses hidden by aggregate accuracy, while the human comparison and exploratory agent experiments provide useful context.

Critical weaknesses

Reviewer follows the statement in section 3.1 to check the details of abalation experiment in the appendix C while C.5 seems missing the results (a broken link). Reviewer cannot verify the authors' statement.

Questions for authors
  1. Can you provide reconciled curation counts, per-instance scores, and the actual generation/judging configurations that reproduce the reported tables?
  2. What independent human validation supports the perception, relation, and reasoning scores, especially when responses contain only a final answer and the documented judge inputs omit images?
4 comments
AuthorsOpenReview ↗

Response to Reviewer fdcg [Part 1/2]

We thank the reviewer for noting that the category breakdowns reveal weaknesses hidden by aggregate accuracy and that the human and agent comparisons add context.

The concerns are about verifiability. In brief: (i) the per-stage curation evidence behind the broken C.5 reference is given in Table R1; (ii) Table 7 is reconciled and every row now closes arithmetically; (iii) the full configuration is in Table R2; (iv) a new human audit with image access validates the stage labels (kappa = 0.84 to 0.91). We have also highlighted all changes in the revised manuscript in blue.


W1. Broken staged-ablation reference in Appendix C.5

The reviewer is correct that app:pipeline_ablation did not resolve. Every filter logs a rejection reason per instance (Appendix C.1), so each stage's contribution is measured directly as the instances it removes for the defect it targets.

Table R1. Per-stage contribution of the curation pipeline (per-instance logs; Table 7).

Stage / filterDefect targetedInputRemovedRate
1a Metadata integrityMissing image, answer not in choices, empty answer50,0005,59811.2%
1b Publication recencyPre-2020 annotation artifacts44,4025501.2%
1c Question lengthUnder-specified or answer-leaking questions43,8528,62019.7%
1d Exact-text dedupIdentical questions across sources35,2328,54924.3%
1e Taxonomy remappingNot mappable to any of 65 tasks26,6831,8536.9%
2a+2b Visual near-duplicatesSame scene across sources (CLIP cos > 0.97)24,8309,03836.4%
2c Domain balancingVisual cluster > 15% of pool (K = 25)15,7922,87818.2%
3 Text rubric R1 to R4Non-spatial, shallow, knowledge-shortcut, unclear12,9143,09824.0%
4 R5 Visual necessityAnswerable without image (blind test)9,8164054.1%
4 R6 CoherenceQuestion does not match image9,4114995.3%
4 R7 Spatial signalImage lacks spatial structure8,9121,07412.1%
Total50,00042,16215.7% retained

Three entries measure quality directly. Stage 2 found that 36.4% of Stage 1 survivors were cross-source near-duplicates, which text deduplication cannot detect. R5 is an empirical blind test: a multimodal model saw each question and its choices without the image, and the 405 items it answered correctly with high confidence were removed. In Stage 3, the four rubrics raised 7,918 flags on 3,098 removed items (2.56 per item), so at least 1,607 (52%) failed two or more criteria independently; the criteria converge on the same items. Per-stage counts plus a no-image control are the standard way to document curated multimodal benchmarks (MMStar [1]; MMMU-Pro [2]); we report both for 11 filters. All rubrics and thresholds are in Appendix C.3 and C.4, prompts in Appendix L, and code in the supplementary package.

Revised text (Appendix C.5): "We validate the pipeline through per-stage measurement: each filter logs its rejection reason per instance, and Table 7 and Table C.6 report the instances each stage removes for the defect it targets, from 50,000 candidates to 7,838 items (15.7% retained), including 36.4% cross-source near-duplicates and 405 language-shortcut items removed by a blind multimodal test." The other three unresolved references (sec, sec:taxonomy, tab:rw_numbers) are also fixed.


Q1. Reconciled counts, per-instance scores, configurations

(a) Reconciled Table 7. Stage 3 scores all four criteria in one LLM call and removes an instance failing any of them (Appendix C.3). There is no sequential pool, so each criterion reports instances flagged, plus the union.

Revised Table 7.

StageStepFlaggedRemainingReduced by
Raw pool50,000
Stage 11a / 1b / 1c / 1d / 1e44,402 / 43,852 / 35,232 / 26,683 / 24,8305,598 / 550 / 8,620 / 8,549 / 1,853
Stage 22a+2b near-duplicates15,7929,038
2c domain clustering12,9142,878
Stage 3R1 / R2 / R3 / R42,772 / 2,432 / 2,710 / 4
R1 to R4 union3,0989,8163,098
Stage 4R5 / R6 / R79,411 / 8,912 / 7,838405 / 499 / 1,074
Final7,83842,162

Each "Reduced by" equals the difference of consecutive "Remaining" values, reductions sum to 42,162 = 50,000 minus 7,838, and each flag count is below the union, as overlapping flags require. Appendix C.3/C.4 headings now read "12,914 to 9,816" and "9,816 to 7,838." The "60K+" in the abstract counts all instances in the selected source subsets combined; Table 7 starts from the 50,000 admitted into the unified image-question-answer schema, consistent with Section 3.1 ("Stage 1 removes about 50%": (50,000 minus 24,830) / 50,000 = 50.3%). Both are now defined in Section 3.1.

AuthorsOpenReview ↗

Response to Reviewer fdcg [Part 2/2]

(b) Per-instance scores. The release contains, for all 16 models, per-instance responses, the judge's JSON (accuracy, refusal, off-topic, element, relation, reasoning scores with justification), and error labels, plus the versioned manifest and scripts regenerating Table 1, Figure 4, and Tables 11 to 13. The evaluation and curation code is already in the supplementary package. Every instance is scored; failed calls are re-submitted until valid (Appendix D.2).

(c) Configuration.

Table R2. Settings for all reported results.

ItemSetting
Answer prompt / temperatureAppendix L verbatim (identical to released templates); choices as context; free-form / 0.1
Max output tokens600 standard; 20,000 for Claude 4.5 Haiku, GPT-4o-mini, GPT-5-mini, o4-mini, Kimi K2.5, GLM-4.5V
JudgeGemini-3.1-Flash-Image-Preview, temperature 0.1, 1,200 tokens; inputs: question with context, gold answer, full response (no image)
ScoringSemantic equivalence to gold, all reported results
Provider / period / costOpenRouter (Aya-Vision-32B via Cohere API) / March to July 2026 / about $1,200 incl. pilots

Model IDs are listed in our response to Reviewer KASd (Q2). The 1,000-token value in Appendices D and F is the harness default; Appendix F now states only these values.

Q2. Human validation of stage scores without images or traces

The scores are output-level annotations of what each response states (Appendix E, Signal S3), and the judge receives the full response text; in practice most models give rationales (Appendix M). Three lines of evidence support their validity.

(1) Direct human audit. Three annotators labeled 140 responses per dimension with the image, question, gold answer, and response visible; the sample is balanced on the judge label (70/70).

Table R3. Judge vs. human labels (n = 140 per dimension, 3 annotators with image access).

DimensionAccuracy vs. majorityKappa vs. majorityMean kappa vs. each annotatorMean pairwise kappa, annotatorsFleiss' kappa
Elements95.7%0.9140.8950.9050.905
Relations93.6%0.8710.8850.8950.895
Reasoning92.1%0.8430.8480.8950.895

Agreement is "almost perfect" [3] and within 0.05 of inter-annotator agreement, although only the humans see the image; disagreements are symmetric (6, 9, 11 of 140), so the judge is neither lenient nor strict.

(2) Dissociation from correctness. Across 112 model x category cells, element and relation scores correlate with accuracy at r = 0.17 and 0.20 (reasoning chain: 0.98); ordering element > relation > reasoning holds in 112 of 112 cells.

(3) Known groups. On all 23,514 human responses, the same judge gives 100.0 / 99.9 / 99.7, against model means of 78.4 / 57.0 / 35.3 and best-model values of 87.3 / 69.6 / 45.8.

Text-only reference-guided judging is standard (MM-Vet [4]; step-level LLM grading in MathVerse [5]). We keep direct answers because CoT changes the construct: it often degrades perception-heavy tasks [6] and helps mainly math and symbolic reasoning [7]. Accuracy itself is validated in Appendix F.2 (175 items; kappa = 1.000 with the majority; annotator pairwise kappa 0.81). Appendix D.5, which predated these studies, now cites F.2 and Table R3.


In summary, Table R1 supplies the per-stage evidence behind C.5, Table 7 is fully reconciled, Table R2 gives the complete configuration, and Table R3 validates the stage labels against humans at kappa = 0.84 to 0.91. We respectfully ask you to consider updating your score in light of this response.


References:

  • [1] Chen et al. Are We on the Right Way for Evaluating Large Vision-Language Models? (MMStar). NeurIPS 2024.
  • [2] Yue et al. MMMU-Pro. ACL 2025.
  • [3] Landis and Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1), 1977.
  • [4] Yu et al. MM-Vet. ICML 2024.
  • [5] Zhang et al. MathVerse. ECCV 2024.
  • [6] Jiang et al. MME-CoT. ICML 2025.
  • [7] Sprague et al. To CoT or not to CoT? ICLR 2025.
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer fdcg,

We have responded to all your questions and concerns, addressed the identified weaknesses, and updated the revised PDF accordingly. We would greatly appreciate it if you could review our responses and consider updating your score accordingly, as only around 26 hours remain in the discussion period.

Thank you for your time and consideration.

Sincerely, Authors

Thank you for your response. However, I consider keep my current score.

Reviewer_KASdOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FrontierSpatial collects 23 existing datasets into a unified benchmark with 7,838 questions and 11,825 images, covering 7 categories, 19 subcategories, and 65 tasks. It evaluates 16 multimodal models, introduces automated scoring and error-analysis pipelines, and tests two agentic systems on a subset of hard cases.

The broad coverage and unified evaluation framework can be useful resources for the community. However, I have major concerns about whether the proposed diagnostic protocol really supports the paper's main conclusions about perception, grounding, and reasoning failures. In addition, some key implementation details in the code/supplementary material are missing, which makes it hard to reproduce the reported analyses.

Critical strengths
  1. Broad and unified evaluation coverage. Combining different spatial tasks into a single framework and evaluation protocol helps reduce fragmentation and makes it easier to compare model strengths and weaknesses. The category-level results give a much clearer picture of model capabilities than a single narrow benchmark.

  2. Attention to benchmark quality and evaluation reliability. The authors perform cross-dataset deduplication, visual-necessity filtering, human baselines, and human checks on answer scoring. These are valuable contributions to the benchmark, as long as the data, annotations, and detailed procedures can be fully verified.

Critical weaknesses
  1. The diagnostic claims are not well supported by the model outputs. The prompt only asks the model for a final one-sentence answer, but the evaluation tries to score spatial element identification, relation identification, and reasoning correctness separately—and even attributes errors to specific stages like perception, grounding, or transformation. A wrong final answer alone does not clearly show which of these intermediate stages actually failed.

  2. The released code does not fully match the reported experiments. The supplement includes an evaluation codebase, but the README (line 42 and line 44) explicitly mentions that some steps differ from the original notebook: prompt templates replace a missing prompt generator, and the failure clustering was changed from using all responses to using only incorrect ones. It is unclear whether the main results in the paper were produced using the original code or these updated procedures.

Questions for authors
  1. Diagnostic validity: What independent evidence shows that the spatial-element, spatial-relation, and reasoning scores (and the assigned failure stages) are accurate? Were these labels checked against human annotations separately from just checking the final correct/incorrect answer? Also, did the judge model used in the reported experiments receive images, intermediate reasoning, or any other information that is not in the provided scoring code?

  2. Implementation provenance: Were the error analyses in the paper produced before or after the clustering fix mentioned in the README? Could you please clarify which exact code version and inference settings were used to generate the main tables (including model-specific token budgets), and whether using the corrected procedure changes any of your main conclusions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer KASd [Part 1/3]

We thank the reviewer for recognizing that a single protocol across spatial tasks "gives a much clearer picture of model capabilities than a single narrow benchmark," and that our deduplication, visual-necessity filtering, human baselines, and human scoring checks are valuable.

The review raises two questions: whether stage-wise scores can be read from direct-answer outputs, and which code version produced the reported analyses. In brief: (i) a new human audit with image access shows the judge's stage labels agree with humans at kappa = 0.84 to 0.91, matching inter-annotator agreement; (ii) two analyses of the submitted results show the stage scores are not restatements of correctness; (iii) all reported results use the procedures described in the paper and the released code. We have also highlighted all changes in the revised manuscript in blue.


W1. Can stage-wise scores be read from one-sentence answers?

We agree that a bare answer token cannot reveal an internal failure stage. The scores are therefore defined, and now described, as output-level annotations: the judge grades whether the entities a response names, the relations it states, and the inference it gives are correct (Appendix E, Signal S3: "element identification," "relationship description," "logical consistency"). They are dissociable from correctness by design, and failure stages are assigned to "the earliest verifiable breakdown step" (Section 4.3). We tested whether these annotations are valid in three ways.

Responses are rarely answer-only. The instruction bounds verbosity, not format. In Appendix M, Claude Haiku 4.5 names the six-pointed star, the vertical fold line, and the overlap before answering; Gemma-4-31B-it, GLM-4.5V, and Kimi K2.5 give multi-step rationales; Gemini-3.1-Flash-Lite states the operative relation in one sentence. The judge receives the full response text.

(1) Direct human audit. Three annotators independently labeled 140 model responses per dimension, seeing the image, question, gold answer, and response. The sample is balanced on the judge label (70 positive, 70 negative), so agreement cannot be inflated by a majority class.

Table R1. Judge vs. human labels for stage annotations (n = 140 per dimension, 3 annotators with image access).

DimensionAccuracy vs. human majorityKappa, judge vs. majorityMean kappa, judge vs. each annotatorMean pairwise kappa, annotatorsFleiss' kappaUnanimous human items
Elements95.7%0.9140.8950.9050.90592.9%
Relations93.6%0.8710.8850.8950.89592.1%
Reasoning92.1%0.8430.8480.8950.89592.1%

Agreement is in the "almost perfect" range [5], and the judge agrees with individual annotators within 0.05 of how annotators agree with each other, although only the humans see the image. Disagreements are few and symmetric (6, 9, 11 of 140; split 3/3, 6/3, 5/6 between judge-positive and judge-negative), so the judge is neither lenient nor strict.

AuthorsOpenReview ↗

Response to Reviewer KASd [Part 2/3]

(2) Dissociation from correctness. If the judge inferred stages from the final answer, element and relation scores would track accuracy. Across the 112 model x category cells (Tables 1, 11 to 13):

Table R2. Stage scores vs. final accuracy (16 models x 7 categories).

ScoreMean (%)Pearson r with accuracyWithin-category r (7 categories)Mean score minus accuracy (points)
Elements78.40.170.20 to 0.69+44.9
Relations57.00.200.10 to 0.70+23.5
Reasoning chain35.30.980.95 to 0.98+1.8
Final accuracy33.51.001.000.0

Elements and relations share at most 4% of their variance with accuracy, while the reasoning chain tracks it (mean absolute difference 2.2 points), as a valid chain annotation should. The ordering element > relation > reasoning holds in 112 of 112 cells. In Quantitative reasoning, for example, models score 75.7 to 88.5% on elements but 8.9 to 18.7% on reasoning: they name the right objects and compute the wrong quantity.

(3) Known groups. All three annotators answered all 7,838 items (about 150 hours each; 23,514 responses), scored by the same judge and prompt.

Table R3. Stage scores of human vs. model responses under the identical judge (Figure 4).

GroupElementsRelationsReasoning
Human (two-of-three majority)100.099.999.7
Best model per dimension87.3 (Claude 4.5 Haiku)69.6 (Kimi K2.5)45.8 (Gemini-3.1-Flash-Lite)
Mean of 16 models78.457.035.3
Weakest model per dimension70.6 (Gemma-3-4B-it)42.4 (Llama-3.2-11B)22.2 (Gemma-3-4B-it)

The judge separates humans from the best model by 12.7 points on elements and 30.3 on relations, and human element/relation scores exceed human accuracy (91.22%), again showing that each dimension is graded on its own content.

Why direct answers rather than chain-of-thought. CoT changes the construct we measure: MME-CoT finds CoT often degrades LMMs on perception-heavy tasks [1], and a meta-analysis of 100+ papers finds CoT gains concentrate on math and symbolic reasoning [2]. Imposing CoT would shift exactly the perception-to-reasoning gap under study. LLM-judged step and error labels are established practice (MathVerse [3]; MME-CoT [1]); we add the three validity checks above.

Revised text (Section 3.2): "Each response is scored on final-answer accuracy and three output-level stage annotations that grade the content the response commits to: the spatial entities it names (elements), the relations it states (relations), and whether its stated inference leads to the gold answer (reasoning). These annotations are dissociable from correctness by design (Appendix E, Signal S3) and are validated against three annotators with image access (kappa = 0.84 to 0.91)."

AuthorsOpenReview ↗

Response to Reviewer KASd [Part 3/3]

W2. Does the released code match the reported experiments?

All reported results use the procedures described in the paper and implemented in the released code; the README notes document development history.

  • Prompt templates (line 42). The released templates are character-identical to Appendix L, the strings used in all runs. The original notebook generated them with a helper not included in the package; no model or judge input changes.
  • Clustering (line 44). The reported error analysis clusters incorrect responses only, as specified in Appendix E (Step 2a embeds "each incorrect item"; incorrect items are sorted into format, reasoning, and knowledge/perception pools before clustering) and Section 3.3. An early notebook version embedded all responses; it was corrected before any reported analysis. The E1 to E7 schema was fixed before model evaluation (Appendix E, Step 2d), and the cluster-to-taxonomy mapping was used only after passing a pre-specified gate of kappa >= 0.75 on a 50-item stratified human sample, with rejected items relabeled (Appendix E.2.4).
  • Scope. Clustering affects only Section 4.3 and Appendix I. Accuracy, stage scores, agent results, and the judge study come from Stage 1 scoring, which neither note touches.

Q1. Independent evidence, human checks, judge inputs

Evidence: Tables R1 to R3. Human checks: accuracy is validated in Appendix F.2 (175 items; judge vs. majority kappa = 1.000; annotator pairwise kappa 0.81); stage labels in Table R1, separately from correctness; failure-stage mapping through the kappa >= 0.75 gate. Judge inputs: the Appendix L prompt fields only (question with context, gold answer, full response text including any visible reasoning); no images, no hidden reasoning tokens, nothing beyond the released scoring code. Reference-guided text-only judging is standard (MM-Vet [4]; MathVerse [3]), and Table R1 confirms empirically that image-seeing annotators agree with the text-only judge.

Q2. Provenance and settings

Reported error analyses were produced after the fix (W2), so Section 4.3 already reflects the corrected procedure.

Table R4. Settings for all reported results.

ItemSetting
Answer prompt / temperatureAppendix L verbatim / 0.1
Max output tokens600 standard; 20,000 for Claude 4.5 Haiku, GPT-4o-mini, GPT-5-mini, o4-mini, Kimi K2.5, GLM-4.5V
JudgeGemini-3.1-Flash-Image-Preview, temperature 0.1, 1,200 tokens, no image
ScoringSemantic equivalence to the gold answer; every instance scored (failed calls re-submitted, Appendix D.2)
Provider / period / costOpenRouter (Aya-Vision-32B via Cohere API) / March to July 2026 / about $1,200 incl. pilots
CodeReleased supplementary package

Model IDs: anthropic/claude-haiku-4.5, google/gemini-2.5-flash, google/gemini-3.1-flash-lite, openai/gpt-4o-mini, openai/gpt-5-mini, openai/o4-mini, google/gemma-3-4b-it, qwen/qwen3-vl-8b-instruct, qwen/qwen3.5-9b, meta-llama/llama-3.2-11b-vision-instruct, google/gemma-4-31b-it, Aya-Vision-32B (Cohere), qwen/qwen3.6-35b-a3b, moonshotai/kimi-k2.5, z-ai/glm-4.5v, mistralai/mistral-small-2603.

The token budget is a ceiling, so a larger value prevents truncated reasoning without affecting models that stop earlier. The 1,000-token value in Appendices D and F is the harness default; Appendix F now states only the values above.


In summary, the stage scores are output-level annotations validated three ways (human audit kappa = 0.84 to 0.91; r = 0.17 and 0.20 with accuracy; 12.7 to 30.3-point human-model separation), and all reported results come from the documented procedures and released code. We respectfully ask you to consider updating your score in light of this response.


References:

  • [1] Jiang et al. MME-CoT. ICML 2025.
  • [2] Sprague et al. To CoT or not to CoT? ICLR 2025.
  • [3] Zhang et al. MathVerse. ECCV 2024.
  • [4] Yu et al. MM-Vet. ICML 2024.
  • [5] Landis and Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1), 1977.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview of the Initial Rebuttal

We thank the AC and reviewers KASd, fdcg, and JA5E for their careful reviews, for recognizing the contributions of FrontierSpatial, and for comments that helped us improve the paper. All changes below are in the revised manuscript.

  • Stage-wise scores validated by humans: three annotators with image access agree with the judge at κ = 0.84 to 0.91, which matches how closely the annotators agree with each other. Element and relation scores share at most 4% of their variance with final correctness, so they measure something separate from it.
  • Curation pipeline fully documented: a new per-stage table, taken from per-instance logs, shows 50,000 instances reduced to 7,838. This includes removing 36.4% of items as cross-source near-duplicates and 405 items that could be answered without the image. Table 7 and the 65-task inventory are now reconciled.
  • Robust aggregation: Table 1 now has a macro-average column. Every model's overall score changes by less than its 95% CI under re-aggregation, and the ranking holds (τ = 0.967).
  • Value of joint evaluation: model rankings shift across reasoning categories (mean τ = 0.54), and 6 of 16 models move five or more positions. We also added a direct comparison with five recent broad spatial benchmarks.
  • Reporting: the paper now states the exact run configuration, the scoring mode, and the full human-baseline protocol (all 7,838 items, 3 annotators), and fixes the inconsistencies the reviewers flagged.

These revisions make the benchmark's measurements more transparent and better validated. We are happy to answer any further questions.

AuthorsOpenReview ↗

Closing Summary of the Discussion

We thank the Area Chair and the reviewers for their time and their engagement with FrontierSpatial. Below is a short summary of the discussion and the resulting revisions.

Strengths noted by the reviewers

  • KASd: a single protocol across spatial tasks "gives a much clearer picture of model capabilities than a single narrow benchmark"; cross-dataset deduplication, visual-necessity filtering, human baselines, and human scoring checks are "valuable contributions."
  • fdcg: category breakdowns "reveal weaknesses hidden by aggregate accuracy," and the human comparison and agent experiments "provide useful context."
  • JA5E: a "practically useful attempt to unify heterogeneous evaluation resources"; a structured, scalable curation and evaluation design; category-level analyses across 16 models and a human baseline; and human validation of automated judging.

Discussion. Reviewer KASd confirmed that our response "resolved most of [their] concerns" and raised their score. We thank KASd for the prompt and constructive engagement.

Main concerns and how we addressed them

  1. Validity of stage-wise scores (KASd, fdcg, JA5E). A new human audit with image access (3 annotators, 140 responses per dimension, balanced on the judge label) gives κ = 0.914 / 0.871 / 0.843 for elements, relations, and reasoning, within 0.05 of inter-annotator agreement. Element and relation scores correlate with accuracy at only r = 0.17 and 0.20. On all 23,514 human responses, the same judge separates humans from the best model by 12.7 and 30.3 points.
  2. Reproducibility and configuration (KASd, fdcg, JA5E). We report the full executed configuration (token budgets, judge, scoring mode, model IDs, provider, period, cost), confirm that all reported analyses use the released code and the incorrect-only clustering procedure, and confirm that every instance is scored for every model.
  3. Curation accounting (fdcg, JA5E). We added a per-stage table from per-instance logs for all 11 filters (50,000 to 7,838; 36.4% cross-source near-duplicates removed; 405 items removed by a blind no-image test) and reconciled Table 7 so that every row closes arithmetically.
  4. Inventory and aggregation (JA5E). Table 5 now lists all 65 tasks (8 rows had been omitted) and matches Table 1 exactly. Table 1 gains a macro-average column. Re-aggregation shifts every model by less than its 95% CI half-width (τ = 0.967), and no conclusion changes.
  5. Positioning (JA5E). We added a direct comparison with SpatialScore, OmniSpatial, Spatial-DISE, MMSI-Bench, and SPAR-Bench. FrontierSpatial contains curated subsets of MMSI-Bench and Spatial-DISE, adds a knowledge-grounded family (2,332 items) and a human baseline on the full benchmark, and reveals ranking instability across reasoning families (mean τ = 0.54; 6 of 16 models shift five or more positions) that single-family benchmarks cannot expose.
  6. Reporting fixes (JA5E, fdcg). We fixed the broken cross-references, the gap sign in Table 2, the Quantitative claim, the bold markers in Table 1, the stale naming, and the leftover human row in Table 12.

All revisions are included in the updated PDF and marked in blue.

We thank the AC and all reviewers again. Their comments made the benchmark's measurements more transparent and better validated.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.