acceptodds
Under review as a conference paper at ICLR 2027

GS-MLLM: Cognitively Inspired Triadic Perception for Multiview Spatial Reasoning

12 views

Abstract

Recent progress in spatial MLLMs increasingly leverages geometric priors from 3D vision models. However, exposing reconstructed geometry to an MLLM does not resolve a fundamental problem: how should geometric evidence be organized for reasoning? Geometry must associate repeated observations across views without erasing view-specific details, while a limited reasoning capacity must preserve both question-relevant evidence and global scene structure. We introduce GS-MLLM, a framework for multimodal spatial reasoning that organizes reconstructed geometry into reliability-aware physical groups and combines question-conditioned local evidence with global scene context. These groups consolidate geometrically consistent observations across views while retaining their source-view evidence. A question-conditioned semantic memory retrieves task-relevant local detail, while a question-independent adaptive spatial prefix summarizes the global scene layout. Position-aware, frame-aligned cross-attention integrates the retrieved geometry into Pixel-token states while preserving the MLLM's pretrained visual-token structure. With Qwen3-VL-4B, GS-MLLM achieves 54.7 on VSI-Bench, surpassing the previous best result by 3.1 points, and delivers a further 2.6-point average gain over GeoThinker across four additional spatial reasoning benchmarks. These results suggest that the key to exploiting 3D priors in MLLMs lies not only in providing geometry, but in structuring it around the physical scene and exposing it at the appropriate level of detail.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Model, Spatial Intelligence, 3D Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dQFQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GS-MLLM organizes multiview geometry into physical groups and combines question-conditioned retrieval with a global spatial prefix. Coordinate-grounded cross-attention integrates this geometry into visual-token states. Across two backbones, the method improves over corresponding GeoThinker variants, with Qwen3-VL-4B gaining 3.1 points on VSI-Bench and 2.6 points on average across four additional benchmarks.

Critical strengths
  1. The architecture addresses a clear representation problem coherently. Consolidating observations across views requires retaining view-specific detail, while question-conditioned selection must preserve access to global context. Physical grouping, local retrieval, and a global prefix provide a plausible organization for addressing these competing requirements.

  2. The empirical results span two backbones and several benchmarks. Both backbones outperform their corresponding GeoThinker variants on VSI-Bench, and the Qwen3-VL-4B version improves on all four additional benchmarks. On SPBench, gains are more consistent for spatial-scene questions than for single-image questions, which is consistent with the intended role of the multiview representation.

  3. The ablations cover the principal components and include sensitivity analyses. Table 3 examines physical grouping, consensus, retrieval, the global prefix, and geometry integration. These results provide initial evidence that the proposed components contribute to performance.

Critical weaknesses
  1. The contribution over the closest prior work is not fully isolated. The design builds on GeoThinker’s active integration and SparseVLM’s text-guided selection intuition. The central question is whether grouping and the local–global organization improve over simpler interfaces under comparable budgets.
  2. Baseline comparability remains an additional concern. The relevant data, frame sampling, resolution, and evaluation settings should be compared explicitly.
  3. Grouping and reliability are validated mainly through downstream task scores. Spatial proximity, feature thresholds, and constrained union–find are intended to associate observations of the same surface, but incorrect associations can merge different objects. Threshold sensitivity and component-removal experiments are useful, yet the paper does not report grouping quality, singleton frequency, supporting-view distributions, or performance under low overlap and low confidence.
  4. The benefits depend on the task, and the trade-offs deserve more explicit discussion. For the 4B model, route planning decreases from 30.9 to 29.4 relative to GeoThinker; for the 3B model, ViewSpatial decreases from 43.9 to 42.1. The 4B CV-Bench result exceeds GeoThinker but remains below the base Qwen3-VL-4B score of 85.6. These outcomes do not undermine the aggregate gains, but they limit claims of uniformly improved spatial capability.
Questions for authors
  1. Were the closest GeoThinker baselines retrained or reevaluated in the same environment? Please compare frame sampling, image resolution, decoding, and scoring settings.
  2. Can you report grouping quality, singleton rates, supporting-view distributions, and characteristic association errors? A shuffled-correspondence control would also help assess the role of meaningful grouping.
7 comments
AuthorsOpenReview ↗

Response to Reviewer dQFQ

Thank you for recognizing the representation design, evaluation across backbones, and component ablations. We address your concerns below.

Weakness 1. Contribution over prior work and comparable budgets

The contribution over the closest prior work is not fully isolated. The design builds on GeoThinker’s active integration and SparseVLM’s text-guided selection intuition. The central question is whether grouping and the local–global organization improve over simpler interfaces under comparable budgets.

Authors’ response.

Thank you for this comment.

  • Technical contribution over GeoThinker and SparseVLM

    • We build on GeoThinker's active geometry integration and SparseVLM's text-guided selection. Physical groups connect cross-view association, question-guided local retrieval, and global scene summarization through a shared representation.
AspectInherited mechanismGS-MLLM contribution
Geometry organizationGeoThinker's frame-strict geometry attentionPhysical groups associate observations across views while retaining source-view patches.
Evidence selectionSparseVLM's text-rater mechanismGroup-level question relevance guides per-view retrieval of source observations.
Geometry accessGeoThinker's active decoder integrationQuestion-guided local geometry and a question-independent global spatial prefix, integrated with 3D-coordinate-grounded visual queries.
  • Component evidence on VSI-Bench and four additional benchmarks

    • The table reports component ablations with Qwen3-VL-4B on VSI-Bench and four additional benchmarks. The full-model reference follows Tables 2–3, and VSI-Bench ablation scores follow Table 3. Avg. is the unweighted mean across all five benchmarks.
VariantVSI-BenchMMSI-BenchViewSpatialCV-BenchMindCube-TinyAvg.
Full GS-MLLM54.6527.7448.0583.5842.6351.33
Without physical grouping52.8026.4046.8982.5342.0050.12
Without cross-view consensus53.1926.6946.8082.6541.3350.13
Without question-aware retrieval53.2626.7546.9582.5541.8150.26
Without spatial prefix51.8226.6746.9382.7141.4349.91
Without raw-patch consolidation54.2026.8646.9482.5741.6250.44
Without dense coordinate queries53.2426.2346.8282.4541.8150.11
  • Removing physical grouping, question-aware retrieval, or the spatial prefix reduces VSI-Bench by 1.85, 1.39, and 2.83 points, respectively. Each removal also reduces scores on all four additional benchmarks, supporting the contribution of grouping and local–global organization.

  • Budget comparability of the additional controls

    • All added controls retain the full model's local retrieval token budget per view, equal to the final visual token count. The question-aware retrieval control removes question guidance while preserving the spatial prefix length.
    • Removing physical grouping assigns each patch to a separate group. On ViewSpatial, the mean spatial prefix length increases from 21.93 to 33.06 tokens, while accuracy decreases from 48.05 to 46.89.
AuthorsOpenReview ↗

Response to Reviewer dQFQ

Weakness 2 and Question 1. Baseline comparability

Baseline comparability remains an additional concern. The relevant data, frame sampling, resolution, and evaluation settings should be compared explicitly.

Were the closest GeoThinker baselines retrained or reevaluated in the same environment? Please compare frame sampling, image resolution, decoding, and scoring settings.

Authors’ response.

Thank you for raising this concern. We reevaluated the official GeoThinker-3B checkpoint and trained GeoThinker-4B using the official code with Qwen3-VL-4B. Both baselines and GS-MLLM were evaluated in the same environment, with matched preprocessing, prompts, decoding, and scoring within each backbone. We will clarify this provenance in the manuscript.

Training configuration

The GeoThinker column below describes our 4B reproduction. The 3B baseline retains the training of the official checkpoint.

To accommodate GPU memory and compute constraints, we enabled depart_smi_token=True when training both GeoThinker-4B and GS-MLLM-4B with the Qwen3-VL-4B backbone.

SettingGeoThinker-4B reproductionGS-MLLM
BackboneQwen3-VL-4B-InstructQwen2.5-VL-3B-Instruct / Qwen3-VL-4B-Instruct
Training data234K SPAR + 64K LLaVA-Hound234K SPAR + 64K LLaVA-Hound
Training viewsAnnotated SPAR views, uniformly sampled video frames with a 4–8-frame budget and 2 s target intervalSame sampling policy
Trainable modulesLLM and geometry-integration modulesLLM and geometry-integration modules
Frozen modulesVision encoder, visual merger, VGGT-1BVision encoder, visual merger, VGGT-1B
OptimizerAdamW, learning rate 1e-5, weight decay 0.01AdamW, learning rate 1e-5, weight decay 0.01
Training schedule1 epoch, cosine decay, 3% warmup1 epoch, cosine decay, 3% warmup
Effective batch size64, from 8 GPUs × 1 sample/GPU × 8 accumulation steps64, from 8 GPUs × 1 sample/GPU × 8 accumulation steps
Precision and hardwareBF16, DeepSpeed ZeRO-2, 8 NVIDIA A800 GPUsBF16, DeepSpeed ZeRO-2, 8 NVIDIA A800 GPUs
Training seed00

Shared evaluation configuration

SettingGeoThinker and GS-MLLM
Evaluation frameworkThe same local LMMs-Eval adapter, task definitions, prompts, and scorers
Video samplingUniformly spaced indices over the full video, up to 32 frames. All frames are used for shorter videos
Image-based benchmarksUse the benchmark-provided image set, including single-image CV-Bench inputs
Visual-processor pixel budget12,544–451,584 pixels per image, with dynamic resolution
Geometry preprocessingResize to width 518 with aspect ratio preserved, round height to a multiple of 14, and center-crop height above 518
4B long-sequence downsampling (depart_smi_token)Both 4B models apply 2×2 visual-token downsampling when the input contains more than 8 images
Evaluation batch size1 sample per GPU
GenerationGreedy decoding, temperature 0, top-p 1.0, one beam, sampling disabled
Output-token limit128 for VSI-Bench and SPBench, 16 for MMSI-Bench, ViewSpatial, CV-Bench, and MindCube-Tiny
Prompt formatMatched backbone-specific chat templates and benchmark instructions requesting an option letter or a short numerical answer
Answer parsingExtract the first space-delimited token and remove a trailing period. Multiple-choice matching is case-insensitive, and numerical answers are converted to floats
VSI-Bench scoringMRA over thresholds 0.50–0.95 in steps of 0.05 for numerical tasks, exact-match accuracy for multiple-choice tasks. Relative direction first averages its three difficulty subsets, followed by an equal-weight mean over all eight task groups
Evaluation environmentPyTorch 2.5.1, Transformers 4.57.0, FlashAttention 2.7.4.post1
AuthorsOpenReview ↗

Response to Reviewer dQFQ

Weakness 3 and Question 2. Grouping quality and reliability

Grouping and reliability are validated mainly through downstream task scores. Spatial proximity, feature thresholds, and constrained union–find are intended to associate observations of the same surface, but incorrect associations can merge different objects. Threshold sensitivity and component-removal experiments are useful, yet the paper does not report grouping quality, singleton frequency, supporting-view distributions, or performance under low overlap and low confidence.

Can you report grouping quality, singleton rates, supporting-view distributions, and characteristic association errors? A shuffled-correspondence control would also help assess the role of meaningful grouping.

Authors’ response.

We thank you for these suggestions.

  1. Grouping quality

    • Evaluation. On 128 ScanNet RGB-D view sets with 2–3 views each, we check ground-truth (GT) patch-center distances for same-group cross-view pairs.
    • Results. At a 10 cm tolerance, geometric precision is 85.14%, versus 0.43% with shuffled correspondences. Recall against GT reciprocal-nearest-neighbor pairs within 10 cm is 48.02%.
  2. Singleton rates and supporting-view distribution

    Across all 288 VSI-Bench videos, singletons constitute 63.37% of valid groups and 37.53% of valid observations. Mean support is 1.689 views per group.

    Supporting views12345–8≥9
    Fraction of valid groups63.37%19.86%8.80%4.17%3.58%0.22%
  3. Characteristic association errors

    • Inspected false matches occur around repeated chairs, chair/table boundaries, and glass/reflective regions.
    • Among pairs failing the 10 cm check, 40.63% involve patches with depth standard deviation above 0.1 m.
  4. Grouping reliability under low overlap and low confidence

    On the same RGB-D view sets, GT overlap measures the fraction of valid patches with a cross-view GT neighbor within 10 cm, averaged bidirectionally over view pairs. We stratify sets by GT overlap or mean raw VGGT confidence. Each quartile contains 32 view sets. The table reports geometric precision at 10 cm.

    StratificationLowest quartileHighest quartile
    GT overlap64.69%89.73%
    Mean raw VGGT confidence78.29%93.17%
  5. Shuffled-correspondence control and low-confidence task performance

    • Control. With a fixed GS-MLLM checkpoint using Qwen3-VL-4B, we shuffle valid patch-to-group assignments independently within each view, preserving group sizes, supporting-view counts, singleton counts, and local retrieval and spatial prefix token budgets.
    • Low-confidence subsets. We select questions in each benchmark's bottom quartile of mean raw VGGT confidence, including ties, using the same question IDs for Full and Shuffled.
    • Full-model reference. Full benchmark scores follow Table 2.
    BenchmarkFullShuffledLow-confidence questionsLow-confidence FullLow-confidence Shuffled
    CV-Bench83.5883.6266076.6276.99
    MindCube-Tiny42.6342.3826647.3746.62
    MMSI-Bench27.7426.2625116.5516.16
    ViewSpatial48.0547.651,43048.6847.82

Comment 4. Task-dependent benefits

The benefits depend on the task, and the trade-offs deserve more explicit discussion. For the 4B model, route planning decreases from 30.9 to 29.4 relative to GeoThinker; for the 3B model, ViewSpatial decreases from 43.9 to 42.1. The 4B CV-Bench result exceeds GeoThinker but remains below the base Qwen3-VL-4B score of 85.6. These outcomes do not undermine the aggregate gains, but they limit claims of uniformly improved spatial capability.

Authors’ response.

We appreciate this observation. We agree that the claims should reflect the observed trade-offs. The 4B route-planning score decreases from 30.9 to 29.4, the 3B ViewSpatial score from 43.9 to 42.1, and the 4B CV-Bench score remains below the base model’s 85.6. Sections 4.2–4.3 discuss this variation. On SPBench, spatial-scene scores improve by 2.49 points with 3B and 4.63 points with 4B, while single-image results are mixed. We will describe the aggregate gains and the spatial-scene improvements on SPBench alongside these task-specific decreases.

Reviewer_qX2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how geometry from a 3D reconstruction model should be organized and exposed to an MLLM for multiview spatial reasoning. It groups geometry patches that correspond across views into shared physical groups, retrieves question-relevant groups, adds a question-independent summary of the scene layout as prefix tokens, and fuses the retrieved geometry into the visual tokens through cross-attention. The method is trained on two Qwen-VL backbones and evaluated on VSI-Bench and several other spatial benchmarks. The paper reports gains over GeoThinker, the closest geometry-based baseline, on all of them.

Critical strengths
  • The design is intuitive. Grouping corresponding observations across views, rather than treating patches independently, is a reasonable unit for selecting geometry. Tab 3 ablates each component, and Tab 7 shows that results are relatively stable across the grouping thresholds.
  • Compared with the backbones it starts from (Qwen2.5-VL-3B-Instruct and Qwen3-VL-4B-Instruct), the gains are large on the multiview benchmarks ViewSpatial and MindCube (about +10 to +16 points, Tab 2), and both backbones improve on the multiview subset of SPBench (Tab 4).
Critical weaknesses

W1. The GeoThinker comparison, which carries the main claims, is not reported reliably.

  • Tab 1, GeoThinker-3B: the row matches, task by task, the ρ=0.75\rho=0.75 row of GeoThinker's fusion-ratio ablation (App Tab 8 of [1], 47.86). It does not match GeoThinker's reported main result (48.9, fusion ratio ρ=0.5\rho=0.5). That is also the setting used for the GeoThinker-7B row, and it is the closest match to GS-MLLM itself, which fuses geometry into the middle half of the decoder layers (Sec 3.4, L242). Against the main result, the 3B gains in Sec 4.2 drop from +1.4 to +0.3 on average and from +9.3 to +2.3 on appearance order.
  • GeoThinker 4B rows and Tab 2 3B rows: no version of [1] (arXiv v1-v5) reports a Qwen3-VL-4B model, or a 3B model on the Tab 2 benchmarks. Yet the GeoThinker rows in Tab 1, Tab 2 and Tab 4 are cited to [1], and App B.1 states that baseline scores follow the original papers (L768). Both gains in the abstract (6.0% and 5.4%) are computed against these rows, and their training setup is not described.
  • Tab 2, missing baseline: Tab 2 copies its other baselines from [1] but leaves out VST-3B-SFT [2], which [1] reports on the same four benchmarks. Its average (about 50.5-50.8, depending on which reported MMSI value is used) is above GS-MLLM-3B (49.28) and level with GS-MLLM-4B (50.50), so the Sec 4.2 claim of the two highest open-source averages depends on this omission. VST is not cited in the paper.

W2. The evidence does not show the gains come from how geometry is organized.

  • Missing base model: Tab 1 omits the backbone Qwen3-VL-4B-Instruct before spatial fine-tuning, although Tab 2 includes it. The Qwen3-VL report [3] gives 59.3 for Qwen3-VL-4B-Instruct on VSI-Bench, above GS-MLLM-4B (54.7). Evaluation protocols might differ (for Qwen3-VL-2B, [3] reports 53.9 while [1] lists 49.4), but this gap is large enough that the base model has to be reported under the paper's own protocol.
  • No geometry-free control: Tab 3 removes one component at a time but never trains the same backbone on the same 298K mixture without geometry. Gains over the base models are concentrated on ViewSpatial and MindCube, while MMSI changes by about 1 point and CV-Bench drops for the 4B model (Tab 2), a pattern that spatial instruction tuning alone could produce.
  • Single runs: App B.2 states that "the wall-clock training time and number of repeated runs remain to be reported with the final experiments" (L785), so neither is given, and the wording suggests the reported results may not be from the final runs. Five of the six ablation gaps in Tab 3 are under 2 points.

W3. Presentation problems make the method hard to follow.

  • Fig 2 is hard to map to Secs 3.2-3.4: decorative elements (a brain, cartoon robots) stand in for the model, and key components (the LLM layers receiving geometry, where the spatial prefix is inserted) are not shown.
  • Its formula for the consensus weight uses a(vi)−1a(vi)-1 where Eq 1 uses na(vi)−1n_{a(vi)}-1, and the figure contains a typo ("Answeer").
  • Backbone labels are inconsistent: "Qwen3-4B" and "Qwen3-VL-4B" refer to the same model, and the short form is defined only in App D.1.

References

[1] Thinking with geometry: Active geometry integration for spatial reasoning, Li et al., ICML 2026

[2] Visual spatial tuning, Yang et al., arXiv 2025

[3] Qwen3-VL technical report, Bai et al., arXiv 2025

Questions for authors
  1. Why does the GeoThinker-3B row in Tab 1 use the ρ=0.75\rho=0.75 ablation row of [1] instead of its reported main result?
  2. How were the GeoThinker Qwen3-VL-4B rows (Tabs 1, 2, 4) and Qwen2.5-VL-3B rows (Tabs 2, 4) obtained? If they were retrained, please give the training data, frame counts and fusion ratio.
  3. What rule decided which baselines appear in Tab 2? Please add VST-3B-SFT [2] or explain why it is excluded.
  4. Please report the backbone Qwen3-VL-4B-Instruct, without your fine-tuning, on VSI-Bench and SPBench under your protocol.
  5. Please report the same backbone fine-tuned on the same 298K mixture without any geometry input.
  6. Are the reported results from the final experiments referred to in App B.2? Please report the training time, and the mean and standard deviation over repeated runs for the main GS-MLLM rows and Tab 3.
  7. Tab 5 and Tab 2 both evaluate on the 1,050-question MindCube-Tiny subset (App B.1, App D.3). In Tab 5, post-QA concatenation scores 59.3-65.9, above GPT-5 and both GS-MLLM models in Tab 2 (42.6 and 50.0). What backbone and training data does Tab 5 use, and does it include MindCube training data? The same 197-token configuration scores 34.0 on VSI-Bench in Tab 6, with object counting at 30.7, below the 62.1 obtained by always giving the most frequent answer (Tab 1, "Chance Level (Frequency)"); how do the authors explain these results?
6 comments
AuthorsOpenReview ↗

Response to Reviewer qX2s

Thank you for recognizing the motivation of physical grouping and the multiview improvements. We address the reporting, experimental, and presentation concerns below.

W1, Q1–Q2. GeoThinker comparison and result provenance

Why does the GeoThinker-3B row in Table 1 use the fusion-ratio ablation result instead of its reported main result?

Authors’ response. We appreciate this question. We used a fusion ratio of 0.75 for both GeoThinker-3B and GeoThinker-4B to maintain a unified hyperparameter configuration across our GeoThinker baseline comparisons in Table 1 and the subsequent benchmarks. We will explicitly document the fusion ratio and shared training and evaluation settings in the table caption and Appendix B.2.

How were the GeoThinker Qwen3-VL-4B rows (Tabs 1, 2, 4) and Qwen2.5-VL-3B rows (Tabs 2, 4) obtained? If they were retrained, please give the training data, frame counts and fusion ratio.

Authors’ response. Thank you for requesting these details.

  • GeoThinker-Qwen3-VL-4B (Tables 1, 2, and 4). We trained GeoThinker on our server using the official public code, with Qwen3-VL-4B-Instruct as the backbone. To accommodate GPU memory and compute constraints, we enabled depart_smi_token=True during training.
  • GeoThinker-Qwen2.5-VL-3B (Tables 2 and 4). We reevaluated the official GeoThinker checkpoint released on Hugging Face, retaining its released weights and original training.

The training settings below apply to our 4B reproduction.

SettingGeoThinker-4B training configuration
Training data234K SPAR + 64K LLaVA-Hound
Training viewsAnnotated SPAR views. Video frames are uniformly sampled with a 4–8-frame budget and a 2 s target interval.
Visual-token downsamplingdepart_smi_token=True, applying 2×2 spatial downsampling when an input contains more than 8 images
OptimizerAdamW, learning rate 1e-5, weight decay 0.01
Schedule1 epoch, cosine decay, 3% warmup
Effective batch size / maximum sequence length64 / 12,800 tokens
Precision and hardwareBF16, DeepSpeed ZeRO-2, 8 NVIDIA A800 GPUs

Both baselines use a fusion ratio of 0.75 and are evaluated alongside GS-MLLM in the same LMMs-Eval environment, with matched preprocessing, prompts, decoding, and scorers within each backbone. Both 4B models use the same visual-token downsampling policy at evaluation. Video evaluation uniformly samples up to 32 frames and uses greedy decoding with batch size one.

W1, Q3. VST baseline

What rule decided which baselines appear in Tab 2? Please add VST-3B-SFT or explain why it is excluded.

Authors’ response. We thank you for this suggestion. We initially excluded VST-3B-SFT to focus on fair comparisons using the SPAR/LLaVA-Hound training recipe shared with GeoThinker and VG-LLM. VST uses a different and substantially larger fine-tuning mixture.

MethodSpatial fine-tuning dataGeneral fine-tuning dataApprox. total
GS-MLLM-3B/4BSPAR (234K)LLaVA-Hound (64K)298K
GeoThinker (official 3B checkpoint / reproduced 4B)SPAR (234K)LLaVA-Hound (64K)298K
VG-LLMSPAR (234K)LLaVA-Hound (64K)298K
VST-3B-SFTVST-P (4.1M)LLaVA-OneVision (2.4M)6.5M

Counts refer to fine-tuning examples.

For completeness, we provide its published results below. Avg. is the unweighted mean of the four benchmarks.

MethodAvg.MMSI-BenchViewSpatialCV-BenchMindCube-Tiny
VST-3B-SFT50.8330.2†52.884.435.9
GS-MLLM-3B49.2827.842.177.250.0
GS-MLLM-4B50.5027.748.183.642.6

VST scores follow GeoThinker’s Table 2, where † denotes evaluation on a reduced MMSI-Bench subset. We will make the training-data differences explicit and qualify the ranking and abstract claims accordingly.

AuthorsOpenReview ↗

Response to Reviewer qX2s

W2, Q4–Q5. Untuned backbone and geometry-free control

Please report the backbone Qwen3-VL-4B-Instruct, without your fine-tuning, on VSI-Bench and SPBench under your protocol.

Please report the same backbone fine-tuned on the same 298K mixture without any geometry input.

Authors’ response. Thank you for requesting these controls. We have evaluated the untuned backbone and trained geometry-free controls on the same 298K mixture.

1. Untuned backbone on VSI-Bench

All runs evaluate 5,130 questions, uniformly sample up to 32 video frames, use a 12,544–451,584-pixel budget per image, and apply greedy decoding. The shared 4B downsampling policy applies 2×2 spatial downsampling when an input contains more than 8 images.

ModelVisual-token policyVSI-Bench
Qwen3-VL-4B-InstructNative grid54.71
Qwen3-VL-4B-InstructShared 4B downsampling52.15
GS-MLLM-4BShared 4B downsampling54.65

GS-MLLM gains 2.50 points over the untuned backbone under the shared downsampling policy. Its score is comparable to the native-grid backbone.

2. Untuned backbone on SPBench

The untuned backbone scores 56.61 overall (51.00 SI, 62.23 SS) on all 1,328 questions. SI contains 1,009 single-image questions, and SS contains 319 multiview questions. GS-MLLM scores follow Table 4.

ModelSI accuracySI MRASS accuracySS MRAAvg.
Qwen3-VL-4B-Instruct42.1458.4342.2073.8754.16
GS-MLLM-4B57.4351.4462.3067.1359.58
GS-MLLM − backbone+15.29−6.99+20.10−6.74+5.42

Avg. is the unweighted mean of the four displayed answer-format metrics.

GS-MLLM improves multiple-choice accuracy on both subsets, with lower numerical MRA. Both evaluations use greedy decoding. Output limits are 32 tokens for the backbone and 128 for GS-MLLM. The backbone run uses a 32,768-token maximum sequence length.

3. Contribution beyond same-data spatial fine-tuning

We fine-tuned the native Qwen backbones for one epoch on 234K SPAR + 64K LLaVA-Hound, with geometry disabled. Within each backbone, evaluation matches GS-MLLM’s preprocessing, sampling, decoding, scoring, and visual-token policy.

ModelVSI-BenchMMSI-BenchViewSpatialCV-BenchMindCube-TinyAvg.
Qwen2.5-VL-3B + same-data SFT45.3428.9041.4978.0046.6748.08
GS-MLLM-3B49.227.842.177.250.049.26
Qwen3-VL-4B + same-data SFT52.3626.7048.4783.8642.2950.74
GS-MLLM-4B54.6527.7448.0583.5842.6351.33

Avg. is the unweighted mean of the five benchmark scores shown. Bold indicates the highest score within each backbone. GS-MLLM scores follow Tables 1–3, using Table 3's two-decimal VSI-Bench score for the 4B model.

The 4B comparison shows an additional 2.29-point VSI-Bench gain beyond same-data SFT, with task-dependent changes across the other benchmarks. For geometry organization specifically, removing physical grouping or the spatial prefix in Table 3 reduces VSI-Bench by 1.85 or 2.83 points, respectively.

AuthorsOpenReview ↗

Response to Reviewer qX2s

W2, Q6. Final runs and variability

Are the reported results from the final experiments referred to in App B.2? Please report the training time, and the mean and standard deviation over repeated runs for the main GS-MLLM rows and Tab 3.

Authors’ response. Thank you for raising this concern.

1. Final checkpoints and training cost

The main GS-MLLM results and Table 3 ablations use final checkpoints with training seed 0. Each training run uses eight NVIDIA A800 GPUs.

BackboneTraining time per runApprox. GPU-hours per run
Qwen2.5-VL-3B16.44 hours131.5
Qwen3-VL-4B14.37 hours115.0

We will update the provisional wording in Appendix B.2 with these completed-run details.

2. Repeated runs and prior reporting practice

Repeating full training across multiple seeds for both backbones would be prohibitively costly within our available compute budget. GeoThinker and VG-LLM present their main comparisons and ablations as point estimates, and VG-LLM explicitly cites computational expense in its statistical-significance checklist.

3. Additional component ablations across benchmarks

We have extended the Qwen3-VL-4B component ablations to MMSI-Bench, ViewSpatial, CV-Bench, and MindCube-Tiny. The full-model reference follows Tables 2–3, and VSI-Bench ablation scores follow Table 3. Avg. is the unweighted mean across all five benchmarks.

VariantVSI-BenchMMSI-BenchViewSpatialCV-BenchMindCube-TinyAvg.
Full GS-MLLM54.6527.7448.0583.5842.6351.33
Without physical grouping52.8026.4046.8982.5342.0050.12
Without cross-view consensus53.1926.6946.8082.6541.3350.13
Without question-aware retrieval53.2626.7546.9582.5541.8150.26
Without spatial prefix51.8226.6746.9382.7141.4349.91
Without raw-patch consolidation54.2026.8646.9482.5741.6250.44
Without dense coordinate queries53.2426.2346.8282.4541.8150.11

Q7. MindCube concatenation results and counting performance

Tab 5 and Tab 2 both evaluate on the 1,050-question MindCube-Tiny subset (App B.1, App D.3). In Tab 5, post-QA concatenation scores 59.3-65.9, above GPT-5 and both GS-MLLM models in Tab 2 (42.6 and 50.0). What backbone and training data does Tab 5 use, and does it include MindCube training data? The same 197-token configuration scores 34.0 on VSI-Bench in Tab 6, with object counting at 30.7, below the 62.1 obtained by always giving the most frequent answer (Tab 1, "Chance Level (Frequency)"); how do the authors explain these results?

Authors’ response. We appreciate these questions.

SettingTable 2Table 5Table 6
Model variantFull GS-MLLMPost-QA concatenation variants197-token post-QA concatenation and full GS-MLLM
BackboneQwen2.5-VL-3B / Qwen3-VL-4BQwen2.5-VL-3BQwen2.5-VL-3B
Training data234K SPAR + 64K LLaVA-HoundMindCube-train only (10K)234K SPAR + 64K LLaVA-Hound
Learning rate1e-51e-41e-5
Epochs1101

The concatenation model appends Spatial and Semantic tokens after the QA tokens, exposing geometry through the backbone’s sequence attention. Full GS-MLLM provides an explicit geometry-to-vision interaction through coordinate-grounded, view-aligned cross-attention and supplies global context through a spatial prefix before the first vision-start token.

  • Table 5 examines which geometry tokens to select. It fixes the post-QA concatenation interface and compares dense VGGT features, object-mask selection, and question-conditioned selection on MindCube-Tiny. Their scores are 59.33, 62.10, and 65.90, respectively, with 197 tokens per view for question-conditioned selection. These results use MindCube-specific supervision.
  • Table 6 examines how to integrate geometry. Its 197-token post-QA concatenation baseline is separately trained on the 298K mixture and scores 34.0 overall and 30.7 on object counting. Full GS-MLLM uses the same Qwen2.5-VL-3B backbone and training recipe, with a different geometry-integration architecture, and scores 49.2 overall and 68.1 on counting, exceeding the 62.1 frequency baseline. Here, 197 denotes the selected geometry tokens per view.

Table 6 compares complete geometry-integration architectures under the same backbone and training recipe.

Reviewer_eF33OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GS-MLLM, a framework for multiview spatial reasoning that combines visual appearance, question semantics, and reconstructed geometry. It organizes geometric observations into reliability-aware physical groups, retrieves question-relevant local features, and constructs a question-independent spatial prefix to summarize global scene context. Retrieved geometry is integrated into visual-token states through coordinate-grounded, view-aligned attention. Experiments across multiple spatial reasoning benchmarks and two MLLM backbones report aggregate improvements over the corresponding GeoThinker variants.

Critical strengths
  1. Relevant motivation. Organizing repeated observations across views while preserving task-relevant details and global context is a meaningful problem for geometry-enhanced MLLMs. The proposed combination of local retrieval and global scene summarization is reasonably motivated.
  2. Broad empirical evaluation. The paper evaluates two MLLM backbones on several benchmarks covering different spatial reasoning capabilities. The reported aggregate improvements over corresponding GeoThinker variants provide evidence of benefits beyond a single backbone or benchmark.
  3. Useful supplementary analysis. The paper includes component ablations, grouping-threshold and prefix-budget sensitivity analyses, and qualitative success and failure cases. These analyses provide useful information about the contributions and behavior of the proposed components.
Critical weaknesses
  1. Motivation. The paper does not show that losing view-specific detail or global context is a major cause of existing models’ errors. The cognitive analogy does little to explain why this design is needed.
  2. Novelty. Selection and fusion largely follow SparseVLM and GeoThinker. The added grouping and reliability rules need stronger justification, and the technical advance beyond combining these components remains unclear.
  3. Experiments. The ablations do not clearly specify replacements or control token budgets. Removing grouping can also change prefix length. Comparisons with simple pooling and question-independent retrieval at the same budget are needed.
  4. Other Issues. The VG-LLM reference has the wrong year and omits an author. Table 5 reports 65.90 on MindCube-Tiny versus 50.0/42.6 in Table 2, without explaining the configuration differences. Fig. 2 incorrectly replaces na(vi)n_{a(vi)} in Eq. (1) with a(vi)a(vi). Figs. 1–4 use raster graphics but not vector graphics, and the overview does not clearly show the main modules.
Questions for authors

See the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eF33

Thank you for recognizing the motivation, cross-backbone evaluation, and supplementary analyses.

Weaknesses 1–2. Motivation and novelty

The paper does not show that losing view-specific detail or global context is a major cause of existing models’ errors. The cognitive analogy does little to explain why this design is needed.

Selection and fusion largely follow SparseVLM and GeoThinker. The added grouping and reliability rules need stronger justification, and the technical advance beyond combining these components remains unclear.

Authors’ response.

Design motivation

GS-MLLM targets multiview spatial reasoning, which requires relating repeated observations while retaining source-view detail and scene-level relationships. Pixel, Semantic, and Spatial representations serve appearance preservation, question-guided evidence access, and scene organization. These requirements motivate local retrieval together with a global scene reference.

Technical contribution

We build on SparseVLM's text raters and GeoThinker's active geometry integration. Our contribution is a physical-group representation shared by cross-view association, relevance scoring, local retrieval, and global-prefix construction. SparseVLM, GeoThinker

AspectInherited mechanismGS-MLLM contribution
Geometry organizationGeoThinker's frame-strict geometry attentionPhysical groups associate observations across views while retaining source patches.
Evidence selectionSparseVLM's text-rater mechanismGroup-level question relevance guides per-view retrieval of source observations.
Geometry accessGeoThinker's active decoder integrationQuestion-guided local geometry and a question-independent global prefix, with 3D-coordinate-grounded visual queries.

Grouping reliability

  • Association constraints. Reciprocal 3D matching, distance and feature-similarity thresholds, and one observation per view restrict candidate associations.
  • Consensus strength. Confidence and supporting-view count scale feature sharing, preserving singleton features.

Tables 3 and 7 provide component and sensitivity evidence. Tables 5 and 6 offer supplementary selection and interface comparisons under their respective training settings. We will explain the cognitive inspiration through these functional roles and qualify causal claims about existing models' errors.

AuthorsOpenReview ↗

Response to Reviewer eF33

Weakness 3. Ablation definitions and token budgets

Ablation replacements and token budgets are unclear. Please compare simple pooling and question-independent retrieval at the same budget.

Authors’ response. We clarify the replacement operations and add matched-budget comparisons with singleton grouping, simple mean pooling, and question-independent retrieval. We report local-retrieval and global-prefix budgets separately.

Matched-budget comparisons

We evaluated the same trained Qwen3-VL-4B checkpoint on all 5,130 VSI-Bench questions. Every variant uses exactly 128 global-prefix tokens per input and 48 or 64 local geometry tokens per view, matched for each input. The runs share the same 32-frame sampling, preprocessing, prompts, batch size, and greedy decoding.

VariantPrefix tokens/inputLocal tokens/viewVSI-BenchNumericalMultiple-choice
Full GS-MLLM12848 / 6454.5661.0948.02
Singleton groups12848 / 6453.8760.4147.32
Uniform mean-pooled global prefix12848 / 6453.7060.2247.16
Question-independent retrieval12848 / 6453.7760.6946.86
  • Singleton grouping treats each raw patch as a separate group.
  • Mean-pooled prefix uniformly partitions valid physical groups in group-index order into 128 nonoverlapping bins and averages their features, 3D coordinates, and confidence within each bin.
  • Question-independent retrieval sets question-derived relevance to zero and ranks local candidates using confidence and supporting-view priors.

At matched token budgets, full GS-MLLM improves VSI-Bench by 0.69, 0.86, and 0.79 percentage points over singleton grouping, uniform mean-pooled global prefix, and question-independent retrieval, respectively. It also achieves the highest Numerical and Multiple-choice scores among these variants. These comparisons evaluate inference-time interventions on one checkpoint.

Fixed-prefix sensitivity

We evaluated one trained Qwen3-VL-4B checkpoint on all 5,130 VSI-Bench questions, assigning each input exactly the specified number of prefix tokens. Grouping, local retrieval, frame sampling, and greedy decoding use the same settings across runs.

Prefix tokens per inputVSI-BenchNumericalMultiple-choice
3253.7460.6246.86
6454.0460.8047.29
12854.5661.0948.02
25654.2360.7647.69

These runs measure prefix-budget sensitivity through inference-time changes to a single checkpoint.

Budget reporting

Tables 5 and 6 provide complementary selection and fusion-interface analyses under their respective training settings. Table 5 uses all tokens for dense VGGT and 197/301 tokens per view for question-conditioned selection.

AuthorsOpenReview ↗

Response to Reviewer eF33

Weakness 4. Reporting and presentation

VG-LLM reference

The VG-LLM reference has the wrong year and omits an author.

Authors’ response. Our citation identifies the intended VG-LLM paper. The entry was imported from Google Scholar, whose BibTeX export lists 2026 and ends the author field with “and others”, rendered as “et al.” by the bibliography style. We will verify the publication metadata against the official proceedings and explicitly enter and display all authors.

@article{zheng2026learning,
  title={Learning from videos for 3d world: Enhancing mllms with 3d vision geometry priors},
  author={Zheng, Duo and Li, Yanyang and Wang, Liwei and others},
  journal={Advances in neural information processing systems},
  volume={38},
  pages={20560--20586},
  year={2026}
}

MindCube-Tiny configurations

Table 5 reports 65.90 on MindCube-Tiny versus 50.0/42.6 in Table 2, without explaining the configuration differences.

Authors’ response. Both tables evaluate the same 1,050-question MindCube-Tiny subset. Their training and integration settings are listed below.

SettingTable 2Table 5
BackboneQwen2.5-VL-3B / Qwen3-VL-4BQwen2.5-VL-3B
Training data234K SPAR + 64K LLaVA-HoundMindCube-train only (10K)
Training epochs110
Learning rate1e-51e-4
Geometry integrationGeometry-to-vision cross-attention + global spatial prefixPost-QA concatenation
MindCube-Tiny accuracy50.0 (3B) / 42.63 (4B)65.90 (197 geometry tokens per view)
  • Table 5 evaluates token selection under MindCube-specific supervision and a fixed concatenation interface.
  • Table 2 evaluates full GS-MLLM after training on the 298K spatial/video instruction mixture.

We will state these configurations in the Table 5 caption and Appendix D.3 so that the training context of each score is explicit.

Figures and notation

Fig. 2 incorrectly replaces na(vi)n_{a(vi)} in Eq. (1) with a(vi)a(vi). Figs. 1–4 use raster graphics but not vector graphics, and the overview does not clearly show the main modules.

Authors’ response.

  • Correct the consensus weight in Figure 2. The expression is

    αvi=c~vina(vi)−1max⁡(V−1,1).\alpha_{vi}=\tilde{c}_{vi}\frac{n_{a(vi)}-1}{\max(V-1,1)}.

    Here, a(vi)a(vi) indexes the physical group and na(vi)n_{a(vi)} counts its distinct supporting views. Equation (1) and the implementation use this supporting-view count.

  • Export Figures 1–4 as vector PDFs. Text, arrows, module boundaries, and plot elements will use vector graphics for readability when enlarged or printed.

  • Clarify Figure 2's computation flow. We will label physical-group construction, question-guided local retrieval, global-prefix construction, and coordinate-grounded, view-aligned fusion, matching Sections 3.2–3.4. The global prefix precedes the first vision-start token. Retrieved local geometry enters visual-token states through cross-attention in the middle decoder layers, after self-attention and before the feed-forward block.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.