- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GS-MLLM organizes multiview geometry into physical groups and combines question-conditioned retrieval with a global spatial prefix. Coordinate-grounded cross-attention integrates this geometry into visual-token states. Across two backbones, the method improves over corresponding GeoThinker variants, with Qwen3-VL-4B gaining 3.1 points on VSI-Bench and 2.6 points on average across four additional benchmarks.
Critical strengths
-
The architecture addresses a clear representation problem coherently. Consolidating observations across views requires retaining view-specific detail, while question-conditioned selection must preserve access to global context. Physical grouping, local retrieval, and a global prefix provide a plausible organization for addressing these competing requirements.
-
The empirical results span two backbones and several benchmarks. Both backbones outperform their corresponding GeoThinker variants on VSI-Bench, and the Qwen3-VL-4B version improves on all four additional benchmarks. On SPBench, gains are more consistent for spatial-scene questions than for single-image questions, which is consistent with the intended role of the multiview representation.
-
The ablations cover the principal components and include sensitivity analyses. Table 3 examines physical grouping, consensus, retrieval, the global prefix, and geometry integration. These results provide initial evidence that the proposed components contribute to performance.
Critical weaknesses
- The contribution over the closest prior work is not fully isolated. The design builds on GeoThinker’s active integration and SparseVLM’s text-guided selection intuition. The central question is whether grouping and the local–global organization improve over simpler interfaces under comparable budgets.
- Baseline comparability remains an additional concern. The relevant data, frame sampling, resolution, and evaluation settings should be compared explicitly.
- Grouping and reliability are validated mainly through downstream task scores. Spatial proximity, feature thresholds, and constrained union–find are intended to associate observations of the same surface, but incorrect associations can merge different objects. Threshold sensitivity and component-removal experiments are useful, yet the paper does not report grouping quality, singleton frequency, supporting-view distributions, or performance under low overlap and low confidence.
- The benefits depend on the task, and the trade-offs deserve more explicit discussion. For the 4B model, route planning decreases from 30.9 to 29.4 relative to GeoThinker; for the 3B model, ViewSpatial decreases from 43.9 to 42.1. The 4B CV-Bench result exceeds GeoThinker but remains below the base Qwen3-VL-4B score of 85.6. These outcomes do not undermine the aggregate gains, but they limit claims of uniformly improved spatial capability.
Questions for authors
- Were the closest GeoThinker baselines retrained or reevaluated in the same environment? Please compare frame sampling, image resolution, decoding, and scoring settings.
- Can you report grouping quality, singleton rates, supporting-view distributions, and characteristic association errors? A shuffled-correspondence control would also help assess the role of meaningful grouping.
7 comments
Response to Reviewer dQFQ
Weakness 2 and Question 1. Baseline comparability
Baseline comparability remains an additional concern. The relevant data, frame sampling, resolution, and evaluation settings should be compared explicitly.
Were the closest GeoThinker baselines retrained or reevaluated in the same environment? Please compare frame sampling, image resolution, decoding, and scoring settings.
Authors’ response.
Thank you for raising this concern. We reevaluated the official GeoThinker-3B checkpoint and trained GeoThinker-4B using the official code with Qwen3-VL-4B. Both baselines and GS-MLLM were evaluated in the same environment, with matched preprocessing, prompts, decoding, and scoring within each backbone. We will clarify this provenance in the manuscript.
Training configuration
The GeoThinker column below describes our 4B reproduction. The 3B baseline retains the training of the official checkpoint.
To accommodate GPU memory and compute constraints, we enabled depart_smi_token=True when training both GeoThinker-4B and GS-MLLM-4B with the Qwen3-VL-4B backbone.
| Setting | GeoThinker-4B reproduction | GS-MLLM |
|---|---|---|
| Backbone | Qwen3-VL-4B-Instruct | Qwen2.5-VL-3B-Instruct / Qwen3-VL-4B-Instruct |
| Training data | 234K SPAR + 64K LLaVA-Hound | 234K SPAR + 64K LLaVA-Hound |
| Training views | Annotated SPAR views, uniformly sampled video frames with a 4–8-frame budget and 2 s target interval | Same sampling policy |
| Trainable modules | LLM and geometry-integration modules | LLM and geometry-integration modules |
| Frozen modules | Vision encoder, visual merger, VGGT-1B | Vision encoder, visual merger, VGGT-1B |
| Optimizer | AdamW, learning rate 1e-5, weight decay 0.01 | AdamW, learning rate 1e-5, weight decay 0.01 |
| Training schedule | 1 epoch, cosine decay, 3% warmup | 1 epoch, cosine decay, 3% warmup |
| Effective batch size | 64, from 8 GPUs × 1 sample/GPU × 8 accumulation steps | 64, from 8 GPUs × 1 sample/GPU × 8 accumulation steps |
| Precision and hardware | BF16, DeepSpeed ZeRO-2, 8 NVIDIA A800 GPUs | BF16, DeepSpeed ZeRO-2, 8 NVIDIA A800 GPUs |
| Training seed | 0 | 0 |
Shared evaluation configuration
| Setting | GeoThinker and GS-MLLM |
|---|---|
| Evaluation framework | The same local LMMs-Eval adapter, task definitions, prompts, and scorers |
| Video sampling | Uniformly spaced indices over the full video, up to 32 frames. All frames are used for shorter videos |
| Image-based benchmarks | Use the benchmark-provided image set, including single-image CV-Bench inputs |
| Visual-processor pixel budget | 12,544–451,584 pixels per image, with dynamic resolution |
| Geometry preprocessing | Resize to width 518 with aspect ratio preserved, round height to a multiple of 14, and center-crop height above 518 |
4B long-sequence downsampling (depart_smi_token) | Both 4B models apply 2×2 visual-token downsampling when the input contains more than 8 images |
| Evaluation batch size | 1 sample per GPU |
| Generation | Greedy decoding, temperature 0, top-p 1.0, one beam, sampling disabled |
| Output-token limit | 128 for VSI-Bench and SPBench, 16 for MMSI-Bench, ViewSpatial, CV-Bench, and MindCube-Tiny |
| Prompt format | Matched backbone-specific chat templates and benchmark instructions requesting an option letter or a short numerical answer |
| Answer parsing | Extract the first space-delimited token and remove a trailing period. Multiple-choice matching is case-insensitive, and numerical answers are converted to floats |
| VSI-Bench scoring | MRA over thresholds 0.50–0.95 in steps of 0.05 for numerical tasks, exact-match accuracy for multiple-choice tasks. Relative direction first averages its three difficulty subsets, followed by an equal-weight mean over all eight task groups |
| Evaluation environment | PyTorch 2.5.1, Transformers 4.57.0, FlashAttention 2.7.4.post1 |
Response to Reviewer dQFQ
Weakness 3 and Question 2. Grouping quality and reliability
Grouping and reliability are validated mainly through downstream task scores. Spatial proximity, feature thresholds, and constrained union–find are intended to associate observations of the same surface, but incorrect associations can merge different objects. Threshold sensitivity and component-removal experiments are useful, yet the paper does not report grouping quality, singleton frequency, supporting-view distributions, or performance under low overlap and low confidence.
Can you report grouping quality, singleton rates, supporting-view distributions, and characteristic association errors? A shuffled-correspondence control would also help assess the role of meaningful grouping.
Authors’ response.
We thank you for these suggestions.
-
Grouping quality
- Evaluation. On 128 ScanNet RGB-D view sets with 2–3 views each, we check ground-truth (GT) patch-center distances for same-group cross-view pairs.
- Results. At a 10 cm tolerance, geometric precision is 85.14%, versus 0.43% with shuffled correspondences. Recall against GT reciprocal-nearest-neighbor pairs within 10 cm is 48.02%.
-
Singleton rates and supporting-view distribution
Across all 288 VSI-Bench videos, singletons constitute 63.37% of valid groups and 37.53% of valid observations. Mean support is 1.689 views per group.
Supporting views 1 2 3 4 5–8 ≥9 Fraction of valid groups 63.37% 19.86% 8.80% 4.17% 3.58% 0.22% -
Characteristic association errors
- Inspected false matches occur around repeated chairs, chair/table boundaries, and glass/reflective regions.
- Among pairs failing the 10 cm check, 40.63% involve patches with depth standard deviation above 0.1 m.
-
Grouping reliability under low overlap and low confidence
On the same RGB-D view sets, GT overlap measures the fraction of valid patches with a cross-view GT neighbor within 10 cm, averaged bidirectionally over view pairs. We stratify sets by GT overlap or mean raw VGGT confidence. Each quartile contains 32 view sets. The table reports geometric precision at 10 cm.
Stratification Lowest quartile Highest quartile GT overlap 64.69% 89.73% Mean raw VGGT confidence 78.29% 93.17% -
Shuffled-correspondence control and low-confidence task performance
- Control. With a fixed GS-MLLM checkpoint using Qwen3-VL-4B, we shuffle valid patch-to-group assignments independently within each view, preserving group sizes, supporting-view counts, singleton counts, and local retrieval and spatial prefix token budgets.
- Low-confidence subsets. We select questions in each benchmark's bottom quartile of mean raw VGGT confidence, including ties, using the same question IDs for Full and Shuffled.
- Full-model reference. Full benchmark scores follow Table 2.
Benchmark Full Shuffled Low-confidence questions Low-confidence Full Low-confidence Shuffled CV-Bench 83.58 83.62 660 76.62 76.99 MindCube-Tiny 42.63 42.38 266 47.37 46.62 MMSI-Bench 27.74 26.26 251 16.55 16.16 ViewSpatial 48.05 47.65 1,430 48.68 47.82
Comment 4. Task-dependent benefits
The benefits depend on the task, and the trade-offs deserve more explicit discussion. For the 4B model, route planning decreases from 30.9 to 29.4 relative to GeoThinker; for the 3B model, ViewSpatial decreases from 43.9 to 42.1. The 4B CV-Bench result exceeds GeoThinker but remains below the base Qwen3-VL-4B score of 85.6. These outcomes do not undermine the aggregate gains, but they limit claims of uniformly improved spatial capability.
Authors’ response.
We appreciate this observation. We agree that the claims should reflect the observed trade-offs. The 4B route-planning score decreases from 30.9 to 29.4, the 3B ViewSpatial score from 43.9 to 42.1, and the 4B CV-Bench score remains below the base model’s 85.6. Sections 4.2–4.3 discuss this variation. On SPBench, spatial-scene scores improve by 2.49 points with 3B and 4.63 points with 4B, while single-image results are mixed. We will describe the aggregate gains and the spatial-scene improvements on SPBench alongside these task-specific decreases.
Response to Reviewer dQFQ
Thank you for recognizing the representation design, evaluation across backbones, and component ablations. We address your concerns below.
Weakness 1. Contribution over prior work and comparable budgets
Authors’ response.
Thank you for this comment.
Technical contribution over GeoThinker and SparseVLM
Component evidence on VSI-Bench and four additional benchmarks
Removing physical grouping, question-aware retrieval, or the spatial prefix reduces VSI-Bench by 1.85, 1.39, and 2.83 points, respectively. Each removal also reduces scores on all four additional benchmarks, supporting the contribution of grouping and local–global organization.
Budget comparability of the additional controls