- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GAMSI augments Qwen3-VL-8B with separate metric and structural query groups, blocks direct attention between these groups, and uses Perceiver-style predictors to align their representations with Depth Anything V2 and VGGT. The experts are used only during training, allowing RGB-only inference. A two-stage curriculum combines spatial instruction tuning with MTS, a collection of 152,776 samples covering 13 tasks from six public datasets. Evaluation spans seven spatial benchmarks and includes a controlled small-scale ablation.
Critical strengths
-
The system combines complementary geometric supervision without retaining the expert models at inference. The broad evaluation and the organization of MTS across single-image, multi-image, and video tasks give the work practical value. The reported seven-benchmark average increases from 67.4 after the first stage to 75.8 after the second stage.
-
Table 2 provides genuine controlled evidence: with the same Qwen3-VL-8B backbone, balanced 10K training subset, and hyperparameters, the average rises from 54.1 for the baseline to 55.0 with structural queries, 55.1 with unmasked dual queries, and 55.7 with masked dual queries. This supports a local benefit from the proposed design.
Critical weaknesses
-
The architectural advance over close prior work is not sufficiently established. VisPer-LM (formerly OLA-VLM; https://arxiv.org/abs/2412.09585) already uses task tokens, task-specific Perceiver embedding predictors, regression and contrastive distillation from multiple visual experts, and teacher removal at inference. GAMSI's depth/structure specialization and backbone attention mask are meaningful differences, but the evidence does not yet establish their significance relative to this template. Table 2 is useful, yet its 1.6-point total gain and 0.6-point mask gain are demonstrated only in the 10K setting. The larger comparisons change training data, stages, or backbones, and therefore do not isolate how much of the full-system gain comes from the architecture rather than the training curriculum. This limits the strength of the central contribution claim.
-
The interpretation of the two pathways exceeds the available evidence. Appendix B names Depth Anything V2 but does not specify the checkpoint or alignment layer, leaving the precise metric supervision unclear. CV-Bench measures relative depth/distance; SPBench does include absolute-distance and object-size tasks, but the reported evidence does not isolate the metric pathway's contribution to these tasks. Likewise, blocking direct query-to-query attention does not establish statistical independence when the representations and model parameters are shared. The proposed separation is plausible, but its claimed functional roles need more direct support.
I recommend weak rejection: the system has practical merits and positive controlled results, but the significance and attribution of its central architectural increment remain insufficiently established.
Questions for authors
-
Contribution and attribution: How does GAMSI differ from the VisPer-LM task-token/predictor template under the same teachers and training budget? Are there existing full-stage ordinary-SFT results with the same backbone, data, and update budget? If not, a limited comparison in the existing 10K setting and a narrower attribution of the full-scale gains would be informative; retraining every large-scale baseline is not requested.
-
Pathway interpretation: Which Depth Anything V2 checkpoint and feature layer are used? Can existing results or a limited fixed-checkpoint pathway intervention distinguish effects on SPBench absolute-scale tasks from relative spatial tasks? Please also delimit what the attention mask guarantees about decoupling.
-
Data clarification: Please identify the original training/evaluation splits, scene or video separation, and duplicate or near-duplicate checks for MTS sources that share dataset families with the evaluation benchmarks. Shared source families alone do not establish contamination; the request is to make the split boundary verifiable.
2 comments
Response to Reviewer oBV2 (2/2)
Q2. Teacher targets, absolute versus relative SPBench tasks, and the mask
(1) Exact teacher targets.
We use the relative-depth Depth Anything V2 ViT-L checkpoint, depth_anything_v2_vitl.pth. The target is the elementwise mean of patch features from zero-based encoder blocks [4, 11, 17, 23], excluding the class token. The student representation is taken from the 35th decoder block (zero-based index 34) and passed through EVG. The structural target is VGGT's final aggregator representation, concatenating frame- and global-attention features along the channel dimension.
Relative-depth geometry, RGB appearance, and metric-valued QA supervision jointly support prediction in physical units. We evaluate this capability through numeric distance answers; CV-Bench’s depth ordering and relative-distance tasks assess relative geometry.
On 1,597 distance questions without a supplied scale reference, the results are:
| Model | Task-macro MRA (%) ↑ | Sample-mean MAE (m) ↓ | RMSE (m) ↓ |
|---|---|---|---|
| Original Qwen3-VL-8B | 33.71 | 0.800 | 1.080 |
| Plain Qwen SFT, S1 | 45.89 | 0.560 | 0.785 |
| GAMSI, S1 | 49.04 | 0.503 | 0.738 |
GAMSI improves MRA over plain SFT by 3.15 points and reduces MAE by 10.2%. MRA is macro-averaged over six distance tasks; MAE and RMSE aggregate samples. Metrics are computed directly from the models' raw distance predictions in meters.
(2) Absolute versus relative SPBench tasks.
In a separate task-level ablation, we evaluate the depth-only, structural-only, and full models on SPBench. Absolute distance and object size use MRA; relative distance and relative direction use accuracy. All values are percentages.
| Benchmark | Configuration | Absolute distance (MRA) | Object size (MRA) | Relative distance (Acc.) | Relative direction (Acc.) |
|---|---|---|---|---|---|
| SPBench-SI | Depth only | 7.65 | 64.15 | 78.02 | 36.60 |
| SPBench-SI | Structural only | 6.98 | 62.46 | 78.02 | 38.24 |
| SPBench-SI | Full model | 10.81 | 66.24 | 71.43 | 40.85 |
| SPBench-MV | Depth only | 36.00 | 68.35 | 100.00 | 27.27 |
| SPBench-MV | Structural only | 34.67 | 66.65 | 82.35 | 36.36 |
| SPBench-MV | Full model | 37.33 | 69.11 | 100.00 | 31.82 |
The full model achieves the highest absolute-distance and object-size MRA on both benchmarks, exceeding structural-only by 3.83 and 3.78 points on SI, and 2.66 and 2.46 points on MV. Depth-only also exceeds structural-only on both absolute-scale tasks.
Relative-task results vary: the full model improves SI direction and matches the best MV distance score, while SI distance and MV direction favor a single pathway. The results support combined pathways for absolute-scale tasks, with overlapping roles across relative tasks. SPBench-MV’s additional object-counting task is outside this four-task breakdown.
(3) Scope of the attention mask.
The mask blocks direct cross-group attention while preserving shared context and backbone parameters; the answer can access both groups. “Decoupled” denotes separate expert supervision and constrained direct interaction between the query groups.
Q3. Verifiable training/evaluation splits and overlap checks
We apologize for omitting the dataset split details from the manuscript. Stage 1 uses the officially released SenseNova-SI-800K training split. MTS contains 152,776 records and 239,008 distinct training images. We checked MTS against all seven evaluation benchmarks as follows:
- Source and scene checks. We normalized available image, video/scan, and scene identifiers within their original dataset namespaces and compared them with the evaluation sets. For ScanNet, we additionally grouped scans by location (
spaceId) to cover different scans of the same physical location. - Image checks. We compared file SHA-256 hashes and hashes of orientation-normalized RGB pixels, including image dimensions. Perceptual hashing and exhaustive CLIP retrieval over the training image library identified additional candidates for visual inspection.
- Image–question checks and findings. We compared complete ordered image inputs together with normalized question stems. These checks found zero overlaps among recoverable source identifiers, zero byte- or pixel-identical training–evaluation images, and zero exact image–question duplicates.
For SPAR-Bench, source-ID metadata was unavailable, so checks relied on image and image–question comparisons and visual screening. We will document the source split names and scene/video grouping rules in the revision.
Response to Reviewer oBV2 (1/2)
Thank you for your careful review and for recognizing the practical value of RGB-only inference, our broad evaluation, and the controlled ablation evidence. We address the architectural contribution, pathway interpretation, and training–evaluation separation below.
W1. Architectural contribution and attribution of the gains
Building on the task-token/predictor template, GAMSI assigns depth and structural supervision to separate query groups, constrains direct cross-group attention, and grounds their representations in frame-level visual features through EVG. Q1 reports the adapted VisPer-LM control and component ablations.
We also added ordinary Qwen3-VL-8B SFT without MSDQ, EVG, or geometric-teacher supervision:
Scores are seven-benchmark macro-averages; the GAMSI values are from Table 1. MTS improves plain SFT by 7.91 points, confirming the contribution of spatial instruction tuning. GAMSI retains an additional 4.53-point advantage after S1 and 5.02-point advantage after S1+S2 through its geometry-aware design and supervision.
W2. Evidence for the pathways and the scope of their interpretation
The depth teacher provides relative-depth features, while metric-valued QA supervision supplies the signal for prediction in physical units. Q2 reports SPBench results by absolute and relative task type and clarifies the scope of the attention mask.
Q1. Comparison with the VisPer-LM template and ordinary SFT
(1) Adapted template and matched components.
We implemented a VisPer-LM-style control using the released TaskTokenResampler, adapted to Qwen3-VL and our depth/VGGT teachers. Both systems use the same teacher caches, student feature layer, predictor dimensions, frozen visual encoder including the merger, and trainable language backbone. The model comparisons in Q1(2) use the 50K training subset, with K=40 per teacher for the dual-teacher configurations.
We apologize for the presentation errors concerning query placement, attention, and feature extraction. The queries follow the user question and precede the assistant header; attention is bidirectional within each query group and blocked directly across groups; EVG uses decoder block 35 (zero-based index 34). We will correct Section 3.1 and Figs. 2–3 accordingly.
The adapted control tests the combined design differences in the table; the no-mask ablation below separately tests masked interaction.
(2) Supplementary comparisons on the 50K subset.
The full-model, depth-only, structural-only, and unmasked scores are means over three training seeds. GAMSI exceeds the adapted VisPer-LM control by 3.17 points, and the depth-only, structural-only, and unmasked variants by 2.46, 1.89, and 2.88 points, respectively. The LM-only-query variant removes expert alignment. The visual-only variant aligns only the LLM-refined visual features through EVG. The learned Q_m and Q_s tokens remain in the LLM input, but their output features are not supplied to EVG. These controls examine the contributions of the two pathways, the attention mask, and query-conditioned grounding.
(3) Ordinary SFT and the training curriculum.
W1 reports the full-stage ordinary-SFT results, separating the benefit of MTS from the additional gains of geometry-aware training.