acceptodds
Desk rejected at ICLR 2027

Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial understanding of the physical world from 2D visual inputs hinges on two complementary forms of geometric knowledge: holistic 3D structural perception and fine-grained metric scale estimation. Existing multimodal large language models (MLLMs) typically address only one facet, ingesting either depth maps or point clouds as additional model inputs, which incurs substantial computational overhead and inherits the generalization limitations of upstream prediction models. We propose GAMSI, a dual-pathway Geometry-Aware MLLM for Spatial Intelligence that takes only RGB images as input while internalizing both forms of geometric prior within a unified autoregressive backbone. Specifically, we introduce Metric-Structure Decoupled Queries (MSDQ) which employ two groups of learnable queries to respectively extract dense metric signals and sparse structural cues from the shared visual context, with a task-decoupled attention mask further preventing the two pathways from contaminating each other. Building on this, an Expert-Guided Visual Grounding (EVG) module projects the aggregated cues back to frame-level visual features and aligns them with vision foundation models, which serve purely as training-time supervision, rather than as model inputs. We further build a multi-task spatial instruction-tuning dataset (MTS) comprising 152,776 samples spanning 13 task types and three visual modalities, consolidated from six public datasets. Trained with a two-stage curriculum, GAMSI achieves state-of-the-art performance on seven spatial intelligence benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language and multimodal language models, Computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_oBV2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GAMSI augments Qwen3-VL-8B with separate metric and structural query groups, blocks direct attention between these groups, and uses Perceiver-style predictors to align their representations with Depth Anything V2 and VGGT. The experts are used only during training, allowing RGB-only inference. A two-stage curriculum combines spatial instruction tuning with MTS, a collection of 152,776 samples covering 13 tasks from six public datasets. Evaluation spans seven spatial benchmarks and includes a controlled small-scale ablation.

Critical strengths
  1. The system combines complementary geometric supervision without retaining the expert models at inference. The broad evaluation and the organization of MTS across single-image, multi-image, and video tasks give the work practical value. The reported seven-benchmark average increases from 67.4 after the first stage to 75.8 after the second stage.

  2. Table 2 provides genuine controlled evidence: with the same Qwen3-VL-8B backbone, balanced 10K training subset, and hyperparameters, the average rises from 54.1 for the baseline to 55.0 with structural queries, 55.1 with unmasked dual queries, and 55.7 with masked dual queries. This supports a local benefit from the proposed design.

Critical weaknesses
  1. The architectural advance over close prior work is not sufficiently established. VisPer-LM (formerly OLA-VLM; https://arxiv.org/abs/2412.09585) already uses task tokens, task-specific Perceiver embedding predictors, regression and contrastive distillation from multiple visual experts, and teacher removal at inference. GAMSI's depth/structure specialization and backbone attention mask are meaningful differences, but the evidence does not yet establish their significance relative to this template. Table 2 is useful, yet its 1.6-point total gain and 0.6-point mask gain are demonstrated only in the 10K setting. The larger comparisons change training data, stages, or backbones, and therefore do not isolate how much of the full-system gain comes from the architecture rather than the training curriculum. This limits the strength of the central contribution claim.

  2. The interpretation of the two pathways exceeds the available evidence. Appendix B names Depth Anything V2 but does not specify the checkpoint or alignment layer, leaving the precise metric supervision unclear. CV-Bench measures relative depth/distance; SPBench does include absolute-distance and object-size tasks, but the reported evidence does not isolate the metric pathway's contribution to these tasks. Likewise, blocking direct query-to-query attention does not establish statistical independence when the representations and model parameters are shared. The proposed separation is plausible, but its claimed functional roles need more direct support.

I recommend weak rejection: the system has practical merits and positive controlled results, but the significance and attribution of its central architectural increment remain insufficiently established.

Questions for authors
  1. Contribution and attribution: How does GAMSI differ from the VisPer-LM task-token/predictor template under the same teachers and training budget? Are there existing full-stage ordinary-SFT results with the same backbone, data, and update budget? If not, a limited comparison in the existing 10K setting and a narrower attribution of the full-scale gains would be informative; retraining every large-scale baseline is not requested.

  2. Pathway interpretation: Which Depth Anything V2 checkpoint and feature layer are used? Can existing results or a limited fixed-checkpoint pathway intervention distinguish effects on SPBench absolute-scale tasks from relative spatial tasks? Please also delimit what the attention mask guarantees about decoupling.

  3. Data clarification: Please identify the original training/evaluation splits, scene or video separation, and duplicate or near-duplicate checks for MTS sources that share dataset families with the evaluation benchmarks. Shared source families alone do not establish contamination; the request is to make the split boundary verifiable.

2 comments
AuthorsOpenReview ↗

Response to Reviewer oBV2 (1/2)

Thank you for your careful review and for recognizing the practical value of RGB-only inference, our broad evaluation, and the controlled ablation evidence. We address the architectural contribution, pathway interpretation, and training–evaluation separation below.

W1. Architectural contribution and attribution of the gains

Building on the task-token/predictor template, GAMSI assigns depth and structural supervision to separate query groups, constrains direct cross-group attention, and grounds their representations in frame-level visual features through EVG. Q1 reports the adapted VisPer-LM control and component ablations.

We also added ordinary Qwen3-VL-8B SFT without MSDQ, EVG, or geometric-teacher supervision:

Training dataPlain Qwen3-VL-8B (%)GAMSI (%)GAMSI advantage (pp)
SenseNova-SI-800K62.8767.4+4.53
SenseNova-SI-800K → MTS70.7875.8+5.02

Scores are seven-benchmark macro-averages; the GAMSI values are from Table 1. MTS improves plain SFT by 7.91 points, confirming the contribution of spatial instruction tuning. GAMSI retains an additional 4.53-point advantage after S1 and 5.02-point advantage after S1+S2 through its geometry-aware design and supervision.

W2. Evidence for the pathways and the scope of their interpretation

The depth teacher provides relative-depth features, while metric-valued QA supervision supplies the signal for prediction in physical units. Q2 reports SPBench results by absolute and relative task type and clarifies the scope of the attention mask.

Q1. Comparison with the VisPer-LM template and ordinary SFT

(1) Adapted template and matched components.

We implemented a VisPer-LM-style control using the released TaskTokenResampler, adapted to Qwen3-VL and our depth/VGGT teachers. Both systems use the same teacher caches, student feature layer, predictor dimensions, frozen visual encoder including the merger, and trainable language backbone. The model comparisons in Q1(2) use the 50K training subset, with K=40 per teacher for the dual-teacher configurations.

Design choiceVisPer-LM-style control used hereGAMSI
Token/predictor parametersA shared learned bank supplies pooled LM task tokens and predictor output latentsGeometry query tokens and separately parameterized EVG output latents
AttentionNative causal attentionBidirectional attention within each query group; direct cross-group attention blocked
Token placementAfter the visual prefixAfter the user question, before the assistant header
Alignment objectiveSmooth L1 + InfoNCEMSE + InfoNCE

We apologize for the presentation errors concerning query placement, attention, and feature extraction. The queries follow the user question and precede the assistant header; attention is bidirectional within each query group and blocked directly across groups; EVG uses decoder block 35 (zero-based index 34). We will correct Section 3.1 and Figs. 2–3 accordingly.

The adapted control tests the combined design differences in the table; the no-mask ablation below separately tests masked interaction.

(2) Supplementary comparisons on the 50K subset.

ConfigurationSeven-benchmark score (%)
VisPer-LM-style/Qwen3-VL adaptation57.52
GAMSI, full model60.69
Depth pathway only58.23
Structural pathway only58.80
Both pathways, no mask57.81
Geometry queries with LM loss only59.21
Visual-only EVG59.08

The full-model, depth-only, structural-only, and unmasked scores are means over three training seeds. GAMSI exceeds the adapted VisPer-LM control by 3.17 points, and the depth-only, structural-only, and unmasked variants by 2.46, 1.89, and 2.88 points, respectively. The LM-only-query variant removes expert alignment. The visual-only variant aligns only the LLM-refined visual features through EVG. The learned Q_m and Q_s tokens remain in the LLM input, but their output features are not supplied to EVG. These controls examine the contributions of the two pathways, the attention mask, and query-conditioned grounding.

(3) Ordinary SFT and the training curriculum.

W1 reports the full-stage ordinary-SFT results, separating the benefit of MTS from the additional gains of geometry-aware training.

AuthorsOpenReview ↗

Response to Reviewer oBV2 (2/2)

Q2. Teacher targets, absolute versus relative SPBench tasks, and the mask

(1) Exact teacher targets.

We use the relative-depth Depth Anything V2 ViT-L checkpoint, depth_anything_v2_vitl.pth. The target is the elementwise mean of patch features from zero-based encoder blocks [4, 11, 17, 23], excluding the class token. The student representation is taken from the 35th decoder block (zero-based index 34) and passed through EVG. The structural target is VGGT's final aggregator representation, concatenating frame- and global-attention features along the channel dimension.

Relative-depth geometry, RGB appearance, and metric-valued QA supervision jointly support prediction in physical units. We evaluate this capability through numeric distance answers; CV-Bench’s depth ordering and relative-distance tasks assess relative geometry.

On 1,597 distance questions without a supplied scale reference, the results are:

ModelTask-macro MRA (%) ↑Sample-mean MAE (m) ↓RMSE (m) ↓
Original Qwen3-VL-8B33.710.8001.080
Plain Qwen SFT, S145.890.5600.785
GAMSI, S149.040.5030.738

GAMSI improves MRA over plain SFT by 3.15 points and reduces MAE by 10.2%. MRA is macro-averaged over six distance tasks; MAE and RMSE aggregate samples. Metrics are computed directly from the models' raw distance predictions in meters.

(2) Absolute versus relative SPBench tasks.

In a separate task-level ablation, we evaluate the depth-only, structural-only, and full models on SPBench. Absolute distance and object size use MRA; relative distance and relative direction use accuracy. All values are percentages.

BenchmarkConfigurationAbsolute distance (MRA)Object size (MRA)Relative distance (Acc.)Relative direction (Acc.)
SPBench-SIDepth only7.6564.1578.0236.60
SPBench-SIStructural only6.9862.4678.0238.24
SPBench-SIFull model10.8166.2471.4340.85
SPBench-MVDepth only36.0068.35100.0027.27
SPBench-MVStructural only34.6766.6582.3536.36
SPBench-MVFull model37.3369.11100.0031.82

The full model achieves the highest absolute-distance and object-size MRA on both benchmarks, exceeding structural-only by 3.83 and 3.78 points on SI, and 2.66 and 2.46 points on MV. Depth-only also exceeds structural-only on both absolute-scale tasks.

Relative-task results vary: the full model improves SI direction and matches the best MV distance score, while SI distance and MV direction favor a single pathway. The results support combined pathways for absolute-scale tasks, with overlapping roles across relative tasks. SPBench-MV’s additional object-counting task is outside this four-task breakdown.

(3) Scope of the attention mask.

The mask blocks direct cross-group attention while preserving shared context and backbone parameters; the answer can access both groups. “Decoupled” denotes separate expert supervision and constrained direct interaction between the query groups.

Q3. Verifiable training/evaluation splits and overlap checks

We apologize for omitting the dataset split details from the manuscript. Stage 1 uses the officially released SenseNova-SI-800K training split. MTS contains 152,776 records and 239,008 distinct training images. We checked MTS against all seven evaluation benchmarks as follows:

  1. Source and scene checks. We normalized available image, video/scan, and scene identifiers within their original dataset namespaces and compared them with the evaluation sets. For ScanNet, we additionally grouped scans by location (spaceId) to cover different scans of the same physical location.
  2. Image checks. We compared file SHA-256 hashes and hashes of orientation-normalized RGB pixels, including image dimensions. Perceptual hashing and exhaustive CLIP retrieval over the training image library identified additional candidates for visual inspection.
  3. Image–question checks and findings. We compared complete ordered image inputs together with normalized question stems. These checks found zero overlaps among recoverable source identifiers, zero byte- or pixel-identical training–evaluation images, and zero exact image–question duplicates.

For SPAR-Bench, source-ID metadata was unavailable, so checks relied on image and image–question comparisons and visual screening. We will document the source split names and scene/video grouping rules in the revision.

Reviewer_fRHkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GAMSI, a spatial-intelligence MLLM built on Qwen3-VL-8B-Instruct that targets two kinds of geometric knowledge: dense metric depth and holistic 3D structure. The model appends two sets of learnable query tokens to the visual tokens, one for metric depth (Q_m) and one for 3D structure (Q_s), and a task-decoupled attention mask prevents Q_s from attending to Q_m. The paper refers to this query design as MSDQ.

The final hidden states of each query group are passed through a Perceiver-style module (EVG), which is supervised with MSE and InfoNCE losses against frozen expert features from Depth Anything (metric pathway) and VGGT (structural pathway). Because the experts and the EVG heads are used only during training, GAMSI requires only RGB input at inference time.

The model is evaluated on seven spatial-intelligence benchmarks (VSI-Bench, MindCube-Tiny, ViewSpatial, CV-Bench, SPBench-SI, SPBench-MV, and SPAR-Bench). It reports a 75.8% macro-average, compared with 64.0% for the strongest competing baseline. Ablations on a 10K-sample subset study the two pathways and the task-decoupled attention mask (Table 2, main paper), as well as the query count K (Table 4, supplementary material).

Critical strengths
  1. Inference-time simplicity. Distilling expert geometry into the backbone during training, with no depth maps, point clouds, or experts needed at test time, is a valuable design goal.
  2. Clear decomposition of the problem. The split into metric and structural questions is intuitive and well illustrated (Fig. 1). Pairing each type of task with a complementary expert (Depth Anything V2 and VGGT) is a smart idea.
  3. Strong results. GAMSI-S1+S2 achieves the best score on all seven benchmarks.
  4. Dataset contribution. MTS covers task types that are under-represented in SenseNova-SI.
  5. Clarity. The paper is generally well written, and the method is easy to follow from Fig. 2 and Section 3
Critical weaknesses
  1. The headline gain cannot be attributed to the architecture. The comparison with the same-backbone baseline, SenseNova-SI-Qwen3-VL-8B (63.4 average), is informative. GAMSI-S1 improves on it by 4.0 points, and Stage 2 adds a further 8.4 points with no architectural change: the only difference between the two stages is the training data. Roughly two-thirds of the final gain over this baseline therefore comes from the MTS data, not from MSDQ or EVG.

MTS includes training sets that directly correspond to four of the seven evaluation benchmarks: MindCube (MindCube-Tiny), SPAR-7M (SPAR-Bench), SpatialLadder-26k (SPBench, which is built with the same pipeline on ScanNet), and VSI-590K (VSI-Bench). Neither SPAR-7M nor SpatialLadder-26k appears in the SenseNova-SI data. The largest Stage-2 gains (+19.0 on MindCube-Tiny, +18.5 on SPBench-SI, +10.3 on SPAR-Bench) fall precisely on these matched benchmarks. The paper attributes this to alignment with its task types, but the same pattern would be expected from in-distribution training alone. It therefore does not support the claim that the Stage-1 priors “are not benchmark-specific.”

The remaining 4.0-point Stage-1 gain is also confounded: GAMSI-S1 is trained for up to 3 epochs on 800K samples with a different recipe, whereas the baseline is trained for 1 epoch on 8M samples. The only controlled evidence for the architecture is the 10K-subset ablation (Table 2), in which MSDQ + EVG improves the average by about 1.6 points.

I ask the authors to provide (a) plain Qwen3-VL-8B fine-tuned with the same recipe on SenseNova-SI-800K, and on SenseNova-SI-800K + MTS, without MSDQ and EVG; and (b) a scene- and question-level overlap analysis between the MTS sources and the test splits.

  1. The ablations are too small and too noisy to support component-level claims. All ablations use a 10K subset (about 1.25% of the Stage-1 data), no seeds or variance are reported, and the differences between variants in Table 2 are at most 1.6 points. Table 4 shows how large the run-to-run variation is: the trend in K is non-monotonic, and the peak at K=40 is driven by a single CV-Bench score (86.3, whereas every other K yields 81–83). Excluding CV-Bench, K=32 to K=56 are tied.

This same run underlies two claims in Sec. 4.3. The first is that the metric pathway “restores” CV-Bench performance (lines 427–430). However, the restored value merely matches the no-query baseline, and every other query configuration scores lower, so this may simply be a favorable run. The second is that removing the mask causes “consistent regressions across most benchmarks” (lines 432–435). In fact, the masked model is worse on 3 of the 7 benchmarks, and once CV-Bench is excluded its advantage nearly disappears.

The structural pathway’s gain on MindCube-Tiny is consistent across all variants and appears robust; my concern is specific to the metric pathway and the mask. I ask the authors to report the mean and standard deviation over several seeds and to run the key ablations at a larger scale.

  1. Key ablations are missing. The two most important are:
  • Queries without EVG. trained with the LM loss only. This would separate expert distillation from the effect of simply adding capacity or register-like tokens.
  • EVG without queries. EVG already takes the LLM-refined patch embeddings f̂_v as input, so a natural baseline aligns f̂_v directly to the expert features. This is close to the 3DRS recipe (Huang et al., 2025), which the paper cites but does not compare against. Further ablations would support the remaining design choices:
  • Q_m only. Table 2 has no metric-only row.
  • Loss components. MSE only vs. InfoNCE only, and sensitivity to λ.
  • Expert assignment. Swapping the experts, or using a single expert for both pathways, would test whether the metric/structural specialization matters.
  • Inference-time query removal. Since the queries precede the question, they cannot depend on it. Dropping or masking them at test time would show whether the answers actually rely on them.
  1. The evidence that the queries carry the geometry is weak Because EVG concatenates the shared patch embeddings f̂_v with the query outputs, the expert features could be predicted mostly from f̂_v, with the queries contributing little. Consequently, the Fig. 5 visualizations do not show that Q_m or Q_s encode depth or structure. The observation that Q_m “does not produce sparse structural artifacts” (line 458) would be expected even without the mask, since each head is trained toward its own target. Stronger evidence would include:
  • linear probes from Q_m and Q_s to both targets, including the cross-targets;
  • a query-shuffling or query-zeroing test inside EVG;
  • quantitative depth error for the decoded depth (e.g., AbsRel or δ1 on NYUv2 or ScanNet).

In addition, per-image colormaps hide exactly the kind of scale errors on which “metric” claims depend. The paper also does not explain how the expert features are decoded into the depth maps and point clouds shown.

  1. The "metric" supervision may not be metric. Depth Anything V2 predicts relative depth by default; its metric variants are separately fine-tuned checkpoints. Moreover, the paper supervises with expert features rather than depth predictions, and DAv2’s encoder features (DINOv2-based) are largely scale-ambiguous. The specific checkpoint and feature layer should be stated, and the claim that this pathway injects absolute scale requires direct evidence.

  2. The structural target is unclear for single images. Part of MTS (and likely a large share of SenseNova-SI) consists of single-image samples. For N=1, VGGT’s inter-frame correspondence and camera-motion signals are trivial, and its output reduces to monocular geometry, much like the depth target. For this portion of the data, the two pathways may therefore supervise largely redundant signals, which undermines the premise that they provide complementary knowledge. The paper should explain what the structural target encodes for single images and report results separately for single- and multi-image inputs.

  3. Minor issues.

  • The question tokens are placed after the queries, so under causal attention the queries are question-agnostic. The paper states that this ordering “does not affect the design,” but it does: it determines whether the queries can condition on the task, and it conflicts with the claim that the queries aggregate “task-relevant context.”
  • “Absolute improvement of 11.8%” should read “11.8 percentage points.”
  • “3DSR” should be “3DRS.”
  • Grouping object counting and appearance order under “structural spatial perception” requires justification.
  • The paper does not state whether the baseline numbers were re-run or copied from prior papers.
Questions for authors
  1. Data-matched control (W1). Can you report plain Qwen3-VL-8B fine-tuned with the same recipe on SenseNova-SI-800K, and on SenseNova-SI-800K + MTS, without MSDQ and EVG? This is the experiment that would most change my assessment.
  2. Overlap and out-of-distribution evaluation (W1). Did you check for scene- or question-level overlap between the MTS sources and the evaluation splits, in particular SpatialLadder-26k vs. SPBench and SPAR-7M vs. SPAR-Bench?
  3. Model selection. How were the final Stage-1 and Stage-2 checkpoints and the query count K selected? Was a held-out validation split used, or the evaluation benchmarks themselves?
  4. Variance (W2). Can you report the mean and standard deviation over several seeds for Table 2, ideally on a larger subset?
  5. Queries vs. alignment (W3, W4). What happens when the queries are trained without EVG, and when EVG is applied directly to f̂_v without queries? Inside EVG, how much does the alignment loss increase if the query outputs are zeroed or shuffled? Does accuracy drop if the queries are removed at inference?
  6. Fig. 5 (W4). How were the depth maps and point clouds in Fig. 5 decoded from the EVG outputs? Can you report quantitative depth error on a standard benchmark?
  7. Expert targets (W5). Which Depth Anything V2 checkpoint (relative or metric) and which feature layer provide the depth targets? Which VGGT features are used for the structural targets?
  8. Single images (W6). What does the VGGT target encode when N=1? Can you break down the results by single-image, multi-image, and video inputs?
  9. Query placement. Why are the queries placed before the question? Did you try placing them after it?
  10. Baseline numbers. Were the baseline scores re-run or copied from prior papers, and under which settings (e.g., the number of frames for VSI-Bench)?
  11. Cost. What are the inference latency and memory overhead of the added queries, the Stage-1 training cost (including expert feature extraction), and the number of trainable parameters?
6 comments
AuthorsOpenReview ↗

Response to Reviewer fRHk (1/5)

Thank you for your careful review and for recognizing the simplicity of RGB-only inference, the clear decomposition of spatial tasks, our strong results, and the contribution of MTS. We address your comments and questions point by point below.

W1. Attribution of the gains and training/evaluation separation

(1) Separating instruction-tuning gains from architectural effects.

We added plain Qwen3-VL-8B fine-tuning without MSDQ, EVG, or geometric-teacher supervision:

ModelS1 mean (%)S1+S2 mean (%)S1 → S1+S2 gain (pp)
Plain Qwen SFT62.8770.78+7.91
GAMSI, paper-reported67.475.8+8.4

GAMSI improves over plain SFT by 4.53 points after S1 and 5.02 points after S1+S2. MTS also improves plain SFT by 7.91 points, separating the benefit of spatial instruction tuning from the additional benefit of geometry-aware training.

(2) Scope of the transfer claim.

We additionally regrouped the existing stage results according to the source relationships highlighted in your review:

Evaluation groupGAMSI S1 → S1+S2 gain (pp)
VSI, MindCube, SPBench-SI/MV, SPAR: equal-weight mean+10.76
ViewSpatialBench+3.40
CV-Bench+1.80

The gains on ViewSpatialBench and CV-Bench extend beyond the five source-associated evaluations. Together with the plain-SFT controls, these results support benefits across spatial tasks and input modalities.

(3) Training/evaluation dataset separation.

We apologize for omitting the dataset split details from the manuscript. Stage 1 uses the officially released SenseNova-SI-800K training split. MTS contains 152,776 records and 239,008 distinct training images. We checked MTS against all seven evaluation benchmarks as follows:

  1. Source and scene checks. We normalized available image, video/scan, and scene identifiers within their original dataset namespaces and compared them with the evaluation sets. For ScanNet, we additionally grouped scans by location (spaceId) to cover different scans of the same physical location.
  2. Image checks. We compared file SHA-256 hashes and hashes of orientation-normalized RGB pixels, including image dimensions. Perceptual hashing and exhaustive CLIP retrieval over the training image library identified additional candidates for visual inspection.
  3. Image–question checks and findings. We compared complete ordered image inputs together with normalized question stems. These checks found zero overlaps among recoverable source identifiers, zero byte- or pixel-identical training–evaluation images, and zero exact image–question duplicates.

For SPAR-Bench, source-ID metadata was unavailable, so checks relied on image and image–question comparisons and visual screening. We will document the source split names and scene/video grouping rules in the revision.

W2. Variance and the strength of the pathway claims

We expanded the pathway ablations to a shared 50K-sample training subset and three independently trained seeds. We report the mean and sample standard deviation across seeds, both across all seven benchmarks and after excluding CV-Bench:

ConfigurationSeven-benchmark mean ± SD (%)Excluding CV-Bench, mean ± SD (%)
Depth only58.23 ± 0.6154.13 ± 0.73
Structural only58.80 ± 0.9654.68 ± 0.81
Both pathways, no mask57.81 ± 1.1454.39 ± 0.84
Full model60.69 ± 1.3356.81 ± 1.36
AuthorsOpenReview ↗

Response to Reviewer fRHk (2/5)

W3. Additional component, loss, and expert-assignment controls

(1) Query, alignment, and pathway controls.

On the 50K training subset, queries trained with the LM loss alone score 59.21%, and visual-only EVG scores 59.08%, compared with the full model’s three-seed mean of 60.69%. The visual-only variant aligns only the LLM-refined visual features through EVG. The learned Q_m and Q_s tokens remain in the LLM input, but their output features are not supplied to EVG. The pathway and masking results are reported in W2.

(2) Losses and expert assignment.

On the 50K training subset, we compared alignment objectives using the same initialization, architecture, teacher targets, LM objective, and fixed 1,400-question evaluation subset.

Alignment objectiveSeven-benchmark subset mean (%)
MSE only47.37
MSE + 0.01 InfoNCE47.86
MSE + 0.1 InfoNCE47.63
MSE + 1 InfoNCE46.25
0.01 InfoNCE only46.94

The default alignment objective is MSE + 0.01 InfoNCE, as specified in Eq. (5). With the original teacher assignment, this setting achieves the highest score among the tested loss settings, improving over MSE-only by 0.49 percentage points. Both small contrastive weights outperform MSE-only.

In a separate teacher-assignment comparison on the same evaluation subset, swapping the teachers improves the mean by 0.61 points relative to the original assignment at λ=0.01, with head capacity, query order, and mask fixed. W2 separately evaluates the contributions of the dual pathways and attention mask.

(3) Related distillation control.

On the 50K training subset, GAMSI’s three-seed mean is 60.69%, exceeding the Qwen3-VL adaptation of 3DRS (58.03%) by 2.66 points. Q5 reports the fixed-checkpoint query interventions.

W4. Evidence that the queries carry geometry

To directly measure geometry encoded in the queries, we added cross-target probes on the frozen Stage-1 checkpoint. Ridge readouts are fitted on 1,600 training-side images, selected on 400 validation images, and evaluated on 800 CV-Bench images:

Teacher-feature targetMetric queries, R²Structural queries, R²Joint queries, R²
Depth Anything V20.2100.3370.376
VGGT−0.0540.1720.224

Joint queries achieve the highest R² for both targets, and depth-related information is accessible from both groups. “Decoupled” refers to separate supervision and the cross-group attention constraint; the groups share visual context and backbone parameters.

R² measures teacher-feature reconstruction relative to a predictor that always outputs the fitting-set mean feature.

W5. Relative-depth supervision and absolute-scale estimation

The teacher is the relative-depth Depth Anything V2 ViT-L checkpoint, depth_anything_v2_vitl.pth. Our target averages patch features from encoder blocks 4, 11, 17, and 23, using zero-based indexing.

The depth teacher supplies depth ordering, object boundaries, and scene-layout cues. Metric-valued QA supervision teaches the mapping from these cues and RGB appearance to distances in meters. Scale references, when supplied in a question, provide an additional cue. The metric pathway thus combines relative-depth feature supervision with metric-valued QA supervision to support estimation in physical units.

We also evaluated numeric distance predictions on 1,597 questions without a supplied scale reference:

ModelTask-macro MRA (%) ↑Sample-mean MAE (m) ↓RMSE (m) ↓
Original Qwen3-VL-8B33.710.8001.080
Plain Qwen SFT, S145.890.5600.785
GAMSI, Stage 149.040.5030.738

Relative to plain SFT, GAMSI improves MRA by 3.15 points and reduces MAE by 10.2%. MRA is macro-averaged over six tasks; MAE and RMSE aggregate samples. Metrics are computed directly from the models' raw distance predictions in meters.

W6. The structural target for single images

For N=1, the VGGT target is the final aggregator representation combining frame- and global-attention features. It provides monocular geometric guidance about the visible scene, including shape, depth layout, and spatial arrangement. With multiple images, it additionally incorporates cross-view context. Q8 reports results by input type.

AuthorsOpenReview ↗

Response to Reviewer fRHk (3/5)

W7. Presentation and task-taxonomy clarifications

We apologize for the presentation errors concerning query placement, attention, and feature extraction. The queries follow the user question and precede the assistant header; attention is bidirectional within each query group and blocked directly across groups; EVG uses decoder block 35 (zero-based index 34). We will correct Section 3.1 and Figs. 2–3 accordingly.

We clarify the implementation and terminology as follows:

ItemClarification
Query placementVisual/question context → learned queries → assistant header → answer.
Improvement from 64.0% to 75.8%11.8 percentage points.
Related-method name3DRS.
Object counting and appearance orderInstance enumeration/cross-frame deduplication and temporal ordering within the broader spatial-reasoning task taxonomy.
Baseline provenanceBaselines were re-run with the released official implementations and EASI evaluation settings; Q10 gives the evaluation configuration.

Q1. Plain Qwen SFT controls at both stages

Please see W1(1) for the two-stage plain-SFT comparison and the corresponding GAMSI gains.

Q2. Scene/question overlap and the scope of OOD claims

W1(3) details the scene, image, and image–question overlap checks, including ScanNet location grouping and the image-based checks for SPAR-Bench. W1(2) reports gains on benchmarks beyond the source-associated group.

Q3. Historical model selection and a new validation-only study

The original final Stage-1 and Stage-2 checkpoints and query count K were selected using the evaluation benchmark scores. Appendix B.3 explicitly reports the original benchmark-based selection of K=40 from {8, 16, 24, 32, 40, 48, 56}.

We completed a new validation-based selection experiment. We trained K=20, 40, and 80 queries per pathway and selected K solely by answer-token NLL on 1,000 held-out training-source questions. The split groups co-occurring images and identical RGB images together; training and validation have no shared image paths, exact RGB images, or exact question-answer pairs.

K per pathwayValidation answer NLL ↓Selected
200.259616No
400.259432Yes
800.260854No

K=40 achieves the lowest validation NLL, supporting the setting used in the paper through an additional selection experiment independent of the evaluation benchmarks.

Q4. Mean and standard deviation across training seeds

Please see W2 for the expanded 50K, three-seed results, including the mean with CV-Bench excluded. The full model exceeds depth-only, structural-only, and unmasked variants by 2.46, 1.89, and 2.88 points across seven benchmarks, and 2.68, 2.13, and 2.42 points after excluding CV-Bench. The aggregate advantage therefore persists after excluding CV-Bench.

Q5. Queries versus alignment: training and inference controls

(1) Retraining controls.

Please see W3(1) for the LM-only-query and visual-only EVG controls.

(2) Query interventions inside EVG.

For the EVG diagnostic, we used a frozen supplementary checkpoint and changed only the query outputs supplied to EVG, holding visual features, teacher targets, and heads fixed. The diagnostic uses 48 training-pool image sequences, with 16 each at N=1, 2, and 4:

InterventionDepth alignment-loss changeVGGT alignment-loss change
Zero query outputs0.067 to 0.0820.066 to 0.165

Zeroing the queries increases both alignment losses, showing that EVG uses query information in this diagnostic.

(3) Answer-side dependence at inference.

Using the frozen Stage-1 checkpoint, we block answer-side attention to selected query keys/values at all decoder layers and evaluate multiple-choice spatial questions:

ConditionForced-choice accuracy (%)
Full model78.38
Block metric queries74.95
Block structural queries76.58
Block both groups74.02

Blocking metric queries, structural queries, or both reduces accuracy by 3.43, 1.80, and 4.36 percentage points, respectively. Both query groups therefore contribute to answer generation on this spatial QA set.

Reviewer_oSfbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The primary contribution of this paper is the proposal of GAMSI, a dual-path, geometry-aware MLLM designed for spatial intelligence; unlike other methods, it relies solely on RGB image inputs during inference.

The innovation of this method primarily lies in the design of "Metric-Structure Decoupled Queries" (Section 3.1) and "Expert-Guided Visual Grounding" (Section 3.2). Specifically, in Section 3.1, the LLM (e.g., Qwen) is tasked with incorporating two sets of learnable queries into the input sequence, with a constraint ensuring that these two sets remain mutually invisible to each other. The innovation in the Expert-Guided Visual Grounding (EVG) component involves concatenating the path features extracted by the LLM with these queries. This process also employs a "Perceiver attention" mechanism, utilizing both MSE and InfoNCE losses to align the features from Depth Anything V2 with those from the Visual Grounding/Grounding (VG) module—though this particular step feels relatively conventional.

Finally, we constructed the MTS dataset; the training results achieved first place across all seven benchmarks. Additionally, ablation studies were conducted using a subset of approximately 10,000 entries.

Critical strengths

The experimental results are truly impressive. The method achieves state-of-the-art (SOTA) performance across seven benchmarks, and the dataset encompasses three modalities: single images, multiple images, and videos—a fact that, to some extent, underscores the authoritative nature of the dataset. I carefully reviewed the experimental results tables, and they appear reliable.

This paper enables reliable spatial awareness using only RGB images. As the authors themselves highlight, the method does not rely on depth maps, point clouds, or external models during inference, resulting in low deployment costs. Furthermore, it clearly distinguishes between the two categories of spatial problems: metric and structural.

Critical weaknesses

I have several concerns regarding the reported performance improvements. First, I question whether the performance gains observed in Stage 2 are due to significant overlap between the training data and the test sets. The MTS dataset incorporates MindCube, SpatialLadder-26k, SPAR-7M (corresponding to SPAR-Bench), and VSI-590K (corresponding to VSI-Bench). This raises the possibility that the improvement in average scores is simply a result of this overlap—a point that requires further clarification.

The notion that a priori approaches do not rely on specific benchmark pairs also requires further clarification. This is crucial for clarity in scientific writing.

Is it necessary to conduct a more appropriate comparison during Stage 1? The most critical baseline should use the same backbone and dataset (800K and 800K+MTS) to perform standard SFT, without including MSDQ/EVG. In my view, only this comparison can clearly elucidate the fundamental reasons for the model's performance improvements.

The experimental design and statistical power of the ablation study are insufficient.

Questions for authors

I have carefully examined the literature review section. A key question that arises is: What are the fundamental differences between MSDQ+EVG and methods such as 3DRS, 3DThinker, Spatial Forcing, and CoVT? Could you compare it with at least one or two of these methods using the same backbone and dataset? Such a comparison would better clarify your work's positioning and novelty.

What are the fundamental methodological differences compared to 3DRS and 3DThinker?

More robust experimental setups are required. For instance, please provide the standard SFT results for the same Qwen3-VL-8B model using the identical dataset (800K samples) but without MSDQ/EVG, and then compare these results—item by item—against those of GAMSI_S1 and GAMSI_S1+S2. I believe this experiment is crucial.

The ablation study would benefit from further improvement. Specifically, please conduct two additional EVG ablations under the same training and evaluation protocols: (i) use only f^v\hat{f}_v, with the query input removed; and (ii) use only Q^\hat{Q}, with f^v\hat{f}_v removed. In addition, could you probe Q^m\hat{Q}_m and Q^s\hat{Q}_s by reporting quantitative depth metrics, including AbsRel and δ1\delta_1, as well as the point-cloud metric Chamfer distance?

Does the paper refer to the relative or metric variant of Depth Anything V2? If it is the relative version, how can you justify the concept of "scale-aware metric priors"?

Can you select K using an independent validation set?

What decoder did you use to generate the depth maps and point clouds in Fig. 5? Did it leverage the frozen head of an expert model?

Additionally, it may be necessary to include separate ablation studies for the MSE and InfoNCE components of the EVG, along with a sensitivity analysis for λ.

4 comments
AuthorsOpenReview ↗

Response to Reviewer oSfb (1/3)

Thank you for your careful review and for recognizing our results across visual modalities and the practical value of RGB-only spatial reasoning.

W1. Training–evaluation separation

We apologize for omitting the dataset split details from the manuscript. Stage 1 uses the officially released SenseNova-SI-800K training split. MTS contains 152,776 records and 239,008 distinct training images. We checked MTS against all seven evaluation benchmarks as follows:

  1. Source and scene checks. We normalized available image, video/scan, and scene identifiers within their original dataset namespaces and compared them with the evaluation sets. For ScanNet, we additionally grouped scans by location (spaceId) to cover different scans of the same physical location.
  2. Image checks. We compared file SHA-256 hashes and hashes of orientation-normalized RGB pixels, including image dimensions. Perceptual hashing and exhaustive CLIP retrieval over the training image library identified additional candidates for visual inspection.
  3. Image–question checks and findings. We compared complete ordered image inputs together with normalized question stems. These checks found zero overlaps among recoverable source identifiers, zero byte- or pixel-identical training–evaluation images, and zero exact image–question duplicates.

For SPAR-Bench, source-ID metadata was unavailable, so checks relied on image and image–question comparisons and visual screening. We will document the source split names and scene/video grouping rules in the revision.

W2. Scope of the geometric priors

The priors are depth and 3D structural representations learned from geometric experts, shared across spatial tasks and input modalities. Stage-2 gains also extend to ViewSpatialBench and CV-Bench, beyond the source-associated benchmarks identified in your review:

Evaluation groupGAMSI S1 (%)GAMSI S1+S2 (%)Gain (pp)
Five source-associated benchmarks, macro-average65.6476.40+10.76
ViewSpatialBench59.663.0+3.4
CV-Bench83.985.7+1.8

The first row averages VSI-Bench, MindCube-Tiny, SPBench-SI, SPBench-MV, and SPAR-Bench. These results show benefits across both evaluation groups. The plain-SFT comparison below separately quantifies GAMSI's advantage when using the same training datasets.

W3. Plain-backbone fine-tuning

We added plain Qwen3-VL-8B fine-tuning without MSDQ, EVG, or geometric-teacher supervision:

Training dataPlain Qwen3-VL-8B (%)GAMSI (%)GAMSI advantage (pp)
SenseNova-SI-800K62.8767.4+4.53
SenseNova-SI-800K → MTS70.7875.8+5.02

Scores are seven-benchmark macro-averages. MTS improves plain SFT by 7.91 points, while GAMSI retains an additional 4.53-point advantage after S1 and 5.02-point advantage after S1+S2. Thus, geometry-aware training provides gains beyond spatial instruction tuning alone.

W4. Larger-scale, multi-seed ablations

We expanded the ablations to a shared 50K-sample subset and three training seeds. Results report the seven-benchmark mean ± sample standard deviation across independently trained models:

ConfigurationAverage ± SD (%)
Depth only58.23 ± 0.61
Structural only58.80 ± 0.96
Both pathways, no mask57.81 ± 1.14
Full model60.69 ± 1.33

The full model improves over depth-only, structural-only, and unmasked dual-pathway variants by 2.46, 1.89, and 2.88 points, respectively, supporting both the dual pathways and the attention mask.

Q1. Positioning and comparison with related methods

(1) What MSDQ+EVG adds.

We apologize for the presentation errors concerning query placement, attention, and feature extraction. The queries follow the user question and precede the assistant header; attention is bidirectional within each query group and blocked directly across groups; EVG uses decoder block 35 (zero-based index 34). We will correct Section 3.1 and Figs. 2–3 accordingly.

GAMSI combines separately supervised depth and structural queries, constrained cross-group attention, and query-conditioned visual grounding. Compared with latent reasoning in 3DThinker and CoVT and action-oriented geometric alignment in Spatial Forcing, GAMSI focuses on separately supervised depth/structure queries for spatial QA. Q2 details the differences from 3DRS and 3DThinker; W4 measures the effects of the dual pathways and mask.

(2) Adapted baseline.

We evaluated a 3DRS adaptation with Qwen3-VL-8B on the 50K training subset:

MethodSeven-benchmark mean (%)
GAMSI (three-seed mean)60.69
3DRS/Qwen3-VL adaptation58.03

GAMSI outperforms the adapted 3DRS baseline by 2.66 percentage points.

AuthorsOpenReview ↗

Response to Reviewer oSfb (2/3)

Q2. Differences from 3DRS and 3DThinker

(1) Compared with 3DRS.

3DRS distills geometric features into visual-token representations. GAMSI introduces two dedicated query groups inside the language backbone, assigns them depth and structural supervision separately, and uses EVG to ground their aggregated information in visual features. W4 quantifies the contributions of the two pathways and their masked interaction.

(2) Compared with 3DThinker.

3DThinker generates 3D latent tokens as intermediate reasoning steps and combines supervised alignment with reinforcement learning. GAMSI learns geometric queries in the input prefix through spatial QA and expert-feature supervision. These queries provide geometric context for answer generation without requiring an explicit latent-reasoning rollout or reinforcement-learning stage.

GAMSI strengthens the geometric representations available for answer generation, while 3DThinker develops an explicit reasoning process over geometric latents.

Q3. Plain SFT at both training stages

W3 reports the two-stage plain-SFT comparison and GAMSI’s 4.53- and 5.02-point advantages on the same training datasets.

Q4. EVG input ablations and quantitative geometry probes

(1) Visual-only and query-only EVG.

We varied the features supplied to EVG on the 50K training subset, retaining the learned Q_m and Q_s tokens in the LLM input in all variants:

EVG inputSeven-benchmark mean (%)
Visual features only59.08
Query features only59.90
Visual + query features (three-seed mean)60.69

Visual-only EVG aligns only the LLM-refined visual features: the Q_m and Q_s tokens remain in the LLM, but their output features are not supplied to EVG. Query-only EVG removes the visual-feature input to EVG. Combining both improves the score by 1.61 and 0.79 points, respectively.

(2) Depth and point-cloud readouts.

We fitted equal-capacity linear readouts from each frozen Stage-1 query group to depth and visible-surface point clouds on NYUv2. The probes use 643 training images, 152 validation images, and all 654 official test images, with training and validation separated by scene. Both query representations have the same 40×4096 input dimensions.

Readout inputDepth AbsRel ↓Depth δ₁ (%) ↑Chamfer distance (m) ↓
Metric queries0.212363.490.2764
Structural queries0.209264.880.2778
Mean prior0.363743.200.5865

Both query groups encode recoverable geometric information: relative to the mean prior, metric queries reduce AbsRel by 41.6% and Chamfer distance by 52.9%; structural queries reduce them by 42.5% and 52.6%, respectively. These probes read the query representations directly.

Q5. Relative-depth teacher and metric estimation

We use the relative-depth Depth Anything V2 ViT-L checkpoint, depth_anything_v2_vitl.pth, averaging patch features from zero-based encoder blocks 4, 11, 17, and 23 as the supervision target.

The depth teacher supplies geometric cues such as depth ordering, object boundaries, and scene layout. Metric-valued QA supervision teaches the mapping from these cues and RGB appearance to physical distances. This combines relative-depth feature supervision with metric-valued QA supervision to support estimation in physical units.

We evaluated numeric distance predictions on 1,597 questions without a supplied scale reference:

ModelTask-macro MRA (%) ↑Sample-mean MAE (m) ↓RMSE (m) ↓
Original Qwen3-VL-8B33.710.8001.080
Plain Qwen SFT, S145.890.5600.785
GAMSI, S149.040.5030.738

GAMSI reduces MAE by 37.1% relative to the original backbone and 10.2% relative to plain SFT, and improves MRA over plain SFT by 3.15 points. Numeric-answer coverage was complete for all three models on this subset. MRA is macro-averaged over six distance tasks; MAE and RMSE aggregate samples. Metrics are computed directly from the models' raw distance predictions in meters.

AuthorsOpenReview ↗

Response to Reviewer oSfb (3/3)

Q6. Independent validation for K

The original manuscript explains the selection in Appendix B.3: we compared K ∈ {8, 16, 24, 32, 40, 48, 56} on the 10K training subset and adopted K=40 because it achieved the highest seven-benchmark macro-average in Table 4.

To additionally select K independently of the evaluation benchmarks, we conducted a validation-based selection experiment using a held-out split of 1,000 questions, excluded from model training. The split was constructed from the training-source datasets, grouping co-occurring images and identical RGB images together before partitioning. Training and validation share no image paths, exact RGB images, or exact question–answer pairs. Evaluation benchmarks were not used for this selection experiment.

Queries per pathway, KValidation answer-token NLL ↓
200.259616
400.259432
800.260854

K=40 achieves the lowest validation NLL, supporting the setting used in the paper.

Q7. Decoders for Fig. 5

Yes, the visualizations use frozen expert decoding heads to map EVG-predicted features into depth maps and point clouds. For depth, the predicted features are supplied to the four feature inputs of the frozen Depth Anything V2 head; its output is normalized per image for color display. The point-cloud visualization uses the frozen VGGT decoding heads and back-projection into 3D.

The teacher outputs provide reference targets, while the GAMSI outputs visualize the student's learned representations through EVG. The expert heads serve as visualization decoders; RGB-only spatial QA uses the trained MLLM with its learned query tokens. Q4 reports geometry recovered directly from the query representations using linear readouts.

Q8. MSE, InfoNCE, and λ sensitivity

On the 50K training subset, we compared EVG alignment objectives using the same initialization, architecture, teacher targets, language-model objective, and fixed 1,400-question evaluation subset.

EVG alignment objectiveλSeven-benchmark subset mean (%)
MSE only047.37
MSE + λ InfoNCE0.0147.86
MSE + λ InfoNCE0.147.63
MSE + λ InfoNCE146.25
InfoNCE only0.0146.94

MSE provides feature-level regression, while InfoNCE encourages correspondence between matched student and teacher representations. Their combination at λ=0.01 achieves the highest score, exceeding MSE-only by 0.49 points and InfoNCE-only by 0.92 points. Both tested small weights, 0.01 and 0.1, improve over MSE-only; increasing λ to 1 reduces performance, supporting a small contrastive weight.

Reviewer_oSfbOpenReview ↗

Acknowledgment of Author Rebuttal and Experimental Additions

Thank you for the comprehensive rebuttal and the extensive additional experiments provided.

The rigorous data-overlap checks (including SHA-256 and perceptual hashing), the controlled plain SFT baselines, and the multi-seed ablation studies adequately address my initial concerns regarding performance attribution, potential data leakage, and the specific contributions of the MSDQ and EVG components.

These clarifications and empirical additions have successfully resolved my questions, and I appreciate the authors' thoroughness in strengthening the paper.

Reviewer_nkDCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes GAMSI, a geometry aware MLLM designed to improve spatial understanding from RGB images. The main idea is to separately model two types of geometric information: metric depth for estimating physical distances and scale, and 3D structural information for understanding scene layout and viewpoint dependent relationships. The authors introduce two groups of learnable queries with an attention mask to separate these representations within a shared language model backbone.

The second component, Expert Guided Visual Grounding (EVG), uses pretrained Depth Anything V2 and VGGT models to supervise the learned representations during training. These external models are not required at inference. The authors also construct MTS, an instruction tuning dataset containing approximately 153K samples across 13 spatial tasks and three input modalities. Using a two stage training pipeline with Qwen3-VL-8B, the final model achieves an average score of ~76% across seven spatial benchmarks, compared with 64.0% for the strongest prior baseline reported in Table 1 which is quite significant.

Critical strengths

The overall empirical results are quite strong, with GAMSI outperforming the compared methods across all seven spatial benchmarks. The improvements on MindCube and SPBench are particularly impressive. I also appreciate that the evaluation covers single images, multiple images and videos, which makes the results more interesting and suggests the method could generalize across different types of spatial understanding tasks.

I also like the idea of transferring geometric knowledge from specialized depth and 3D reconstruction models into an MLLM without needing these models at inference. Combining metric and structural supervision makes sense since they capture different aspects of spatial understanding. While the general idea of geometric distillation is not new, bringing these two sources of supervision together within a single RGB only model has practical value, especially if the inference efficiency benefits can be demonstrated.

Critical weaknesses

My biggest concern is how much of the improvement actually comes from the proposed architecture rather than the additional spatial training data. The model shows substantial improvements after the second training stage, which suggests that the instruction tuning data plays a significant role. However, the paper does not include a baseline where the original Qwen3-VL model is trained on exactly the same data with the same training schedule but without MSDQ and EVG. The ablation results are also not particularly convincing with relatively modest improvements from the full architecture and even smaller gains from the proposed attention mask. Some benchmarks also regress with the mask. I am also a bit concerned about the evaluation setup since several datasets used in MTS are closely related to the evaluation benchmarks, including MindCube, SPAR and SpatialLadder. It is important the author include clear explanation of how the splits were handled to ensure no leakage.

I am also not fully convinced by the novelty and the claims around disentangling the two geometric representations. Prior works such as 3DRS and 3D Aware VLM Fine Tuning with Geometric Distillation have already explored transferring geometric information from pretrained 3D models into vision language models. The dual query design is a reasonable extension, but the contribution feels somewhat incremental. Also, blocking direct attention between the two query groups does not necessarily mean they learn independent or complementary representations since they still share the same backbone and visual features. The qualitative visualizations are interesting but not sufficient to demonstrate this. I would have liked to see more quantitative evidence showing what each pathway actually learns, particularly whether the metric pathway preserves accurate absolute scale rather than mainly relative depth information.

Questions for authors

Can the authors provide a baseline where the original Qwen3-VL backbone is trained on exactly the same datasets and training schedule but without MSDQ and EVG? This would help understand how much of the reported improvement comes from the proposed architecture rather than the additional spatial training data.

How do the authors ensure that MTS and SenseNova-SI training samples do not overlap with the evaluation benchmarks at the image, scene, video or question level? This is particularly important for benchmarks whose related datasets are used during training. Can the authors clarify the exact split selection and filtering procedure?

The ablation study shows relatively modest improvements from the dual pathway architecture, with even smaller gains from the attention mask. Can the authors provide stronger evidence that separating metric and structural knowledge is important? It would be useful to see more comprehensive ablations comparing metric only, structural only, joint queries without masking, and the proposed method. Also, is there quantitative evidence that the two query groups actually encode complementary geometric information?

Which exact Depth Anything checkpoint is used for supervision? Some Depth Anything models estimate relative rather than metric depth. How is absolute metric scale preserved through feature alignment, and can the authors provide quantitative depth or distance estimation results? It would also help to clarify how the depth maps and point clouds in the qualitative visualizations are generated from the learned query representations.

The paper claims reduced inference overhead by removing external geometric models, but the proposed architecture also introduces additional query tokens and computation within the LLM. Can the authors provide measured inference latency, FLOPs and memory usage relative to the original Qwen3-VL backbone and representative methods using external geometric inputs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer nkDC (1/3)

Thank you for your careful review and for recognizing our results across visual modalities and the practical value of RGB-only spatial reasoning.

W1. Contributions of the training data and the proposed method

(1) Plain-backbone fine-tuning.

We added plain Qwen3-VL-8B fine-tuning baselines on the two-stage training data, without MSDQ, EVG, or geometric-teacher supervision:

Training stageTraining dataPlain Qwen3-VL-8BGAMSIGAMSI advantage
S1SenseNova-SI-800K62.87%67.4%+4.53 pp
S1+S2SenseNova-SI-800K → MTS70.78%75.8%+5.02 pp

MTS improves plain Qwen by 7.91 percentage points, confirming the value of spatial instruction tuning. GAMSI provides a further 4.53-point advantage after S1 and 5.02-point advantage after S1+S2 through its geometry-aware design and supervision.

(2) Dual pathways and attention mask.

We expanded the ablation study to a shared 50K-sample training subset and three seeds, reporting the seven-benchmark mean ± sample SD across seeds:

ConfigurationAverage ± SD (%)
Depth only58.23 ± 0.61
Structural only58.80 ± 0.96
Both pathways, no mask57.81 ± 1.14
Full model60.69 ± 1.33

The full model improves over the depth-only and structural-only variants by 2.46 and 1.89 percentage points, respectively, and over the unmasked dual-pathway variant by 2.88 points, supporting both the dual pathways and the mask.

(3) Training/evaluation separation.

We apologize for omitting the dataset split details from the manuscript. Stage 1 uses the officially released SenseNova-SI-800K training split. MTS contains 152,776 records and 239,008 distinct training images. We checked MTS against all seven evaluation benchmarks as follows:

  1. Source and scene checks. We normalized available image, video/scan, and scene identifiers within their original dataset namespaces and compared them with the evaluation sets. For ScanNet, we additionally grouped scans by location (spaceId) to cover different scans of the same physical location.
  2. Image checks. We compared file SHA-256 hashes and hashes of orientation-normalized RGB pixels, including image dimensions. Perceptual hashing and exhaustive CLIP retrieval over the training image library identified additional candidates for visual inspection.
  3. Image–question checks and findings. We compared complete ordered image inputs together with normalized question stems. These checks found zero overlaps among recoverable source identifiers, zero byte- or pixel-identical training–evaluation images, and zero exact image–question duplicates.

For SPAR-Bench, source-ID metadata was unavailable, so checks relied on image and image–question comparisons and visual screening. We will document the source split names and scene/video grouping rules in the revision.

W2. Novelty and interpretation of the geometric representations

We apologize for the presentation errors concerning query placement, attention, and feature extraction. The queries follow the user question and precede the assistant header; attention is bidirectional within each query group and blocked directly across groups; EVG uses decoder block 35 (zero-based index 34). We will correct Section 3.1 and Figs. 2–3 accordingly.

Building on geometric distillation, GAMSI assigns per-frame depth and joint-view geometry to separate query groups and EVG predictors, with an attention mask controlling direct cross-group communication. We compared this design with Qwen3-VL adaptations of 3DRS and the VisPer-LM template on the 50K training subset:

ComparisonGAMSI three-seed mean (%)Adapted control score (%)
3DRS/Qwen3-VL adaptation60.6958.03
VisPer-LM-style/Qwen3-VL adaptation60.6957.52

GAMSI improves over these adapted controls by 2.66 and 3.17 percentage points, respectively.

Q1. Plain Qwen3-VL fine-tuning baseline

Please see W1(1) for the added plain Qwen3-VL-8B baselines and the comparisons after each training stage.

Q2. Training/evaluation separation and overlap checks

(1) Stage-1 data.

We use the officially released SenseNova-SI-800K training data and retain its original training split.

(2) MTS overlap checks.

W1(3) details the MTS overlap checks. We will include the source split names and scene/video grouping rules in the revision.

AuthorsOpenReview ↗

Response to Reviewer nkDC (2/3)

Q3. Pathway ablations and query complementarity

(1) Pathway and mask ablations.

Please see W1(2) for the expanded ablations covering all four requested configurations.

(2) Geometric feature probes.

“Decoupled” refers to the cross-group attention constraint and separate supervision targets; the groups share visual context and backbone parameters.

Using representations from the frozen Stage-1 checkpoint, we fitted ridge probes on 1,600 training-side images, selected them using 400 validation images, and evaluated teacher-feature reconstruction on 800 CV-Bench images:

Teacher-feature targetMetric queries, R²Structural queries, R²Joint queries, R²
Depth Anything V20.2100.3370.376
VGGT−0.0540.1720.224

The joint representation yields the highest reconstruction R² for both targets, while depth-related information is accessible from both query groups.

(3) Query contributions to answer generation.

Using the frozen Stage-1 checkpoint, we blocked answer-side attention to selected query keys/values at all decoder layers and evaluated multiple-choice spatial questions:

Inference configurationForced-choice accuracy (%)
Full model78.38
Metric queries blocked74.95
Structural queries blocked76.58

Blocking metric or structural queries reduces accuracy by 3.43 or 1.80 percentage points, respectively, showing that both groups contribute to answer generation on this spatial QA set.

Q4. Depth supervision, absolute scale, and visualization

(1) Teacher checkpoint and supervision targets.

We use the relative-depth Depth Anything V2 ViT-L checkpoint, depth_anything_v2_vitl.pth. The supervision target averages patch features from zero-based encoder blocks 4, 11, 17, and 23. For VGGT, we use the final aggregator representation, which combines frame- and global-attention features.

The depth teacher supplies features encoding depth ordering and scene geometry, while metric-valued QA supervision teaches the mapping from these cues and RGB appearance to physical distances. Scale references, when supplied in a question, provide an additional cue.

(2) Quantitative distance estimation.

We evaluated numeric answers in meters on 3,021 depth/distance questions using the Stage-1 checkpoint under RGB-only QA inference. The primary subset comprises 1,597 distance questions without a supplied scale reference:

ModelTask-macro MRA (%) ↑Sample-mean MAE (m) ↓RMSE (m) ↓
Original Qwen3-VL-8B33.710.8001.080
Plain Qwen SFT, S145.890.5600.785
GAMSI, Stage 149.040.5030.738

GAMSI reduces MAE by 37.1% relative to the original backbone and 10.2% relative to plain SFT, while improving MRA over plain SFT by 3.15 percentage points. Numeric-answer coverage was complete for all three models on this subset. MRA averages the benchmark’s relative-error threshold scores and is macro-averaged over six tasks; MAE and RMSE aggregate samples. Metrics are computed directly from the models' raw distance predictions in meters.

For the remaining questions with supplied scale references, sample-weighted MRA is 61.31% for GAMSI and 62.28% for plain SFT on 680 absolute-depth questions, and 20.54% for both models on 744 depth-difference questions.

Blocking metric queries on the primary distance subset changes MRA from 49.04% to 48.94%, a 0.10-point difference with a paired 95% interval of [−0.32, +0.54]. This intervention concerns absolute-distance prediction; the larger accuracy drop in Q3 concerns broader multiple-choice spatial tasks.

(3) Depth and point-cloud visualization.

Fig. 5 compares teacher-generated reference targets with GAMSI predictions decoded from EVG features using frozen expert heads. The depth predictions use the Depth Anything V2 head, with per-image normalization for color display; point clouds use VGGT decoding heads and 3D back-projection. The expert heads serve as visualization decoders; RGB-only spatial QA uses the trained MLLM with its learned query tokens.

AuthorsOpenReview ↗

Response to Reviewer nkDC (3/3)

Q5. Measured inference latency, FLOPs, and memory

(1) Multi-view inference costs.

On one H100 with BF16, batch size 1, and KV caching, we measured model inference using sixteen 448×448 RGB frames and 32 output tokens. Qwen and GAMSI process frames as images; Spatial-MLLM uses its native video pipeline with online VGGT:

ModelFirst-token latency (ms)Full generation (ms)Estimated total TFLOPsPeak allocated memory (GiB)
Qwen3-VL-8B185.92983.8463.48517.297
GAMSI, Stage 1198.78996.7765.05617.331
Spatial-MLLM354.981093.8584.87911.294

GAMSI reduces first-token latency by 44.0%, full generation latency by 8.9%, and estimated FLOPs by 23.4% relative to Spatial-MLLM in this setting. Timings are medians over three scan sequences and five repetitions each, excluding CPU preprocessing, loading, and networking.

(2) Overhead relative to Qwen3-VL-8B.

Relative to the original Qwen3-VL-8B backbone, the 80 additional queries increase full generation latency by 1.3% and peak allocated memory by 0.20% (approximately 35 MiB). First-token latency increases by 12.86 ms.

(3) High-resolution inputs.

In a separate native-video evaluation with eight 896×896 frames and 32 output tokens, GAMSI reduces first-token latency from 910.48 to 199.18 ms (78.1%), full generation latency by 39.0%, and estimated FLOPs by 71.9% relative to Spatial-MLLM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.