acceptodds
Under review as a conference paper at ICLR 2027

In3D-VLM: Faithful Integration of Explicit 3D Space for Metric Spatial Reasoning

Abstract

As spatial reasoning becomes increasingly important for vision-language models (VLMs), incorporating explicit 3D geometry has attracted growing attention. However, existing approaches either rely on computationally expensive geometry encoders or mix compressed geometry with RGB features without ensuring that the injected 3D information remains accessible after fusion. Consequently, even when metric 3D observations are available, VLMs may struggle to accurately infer quantitative properties such as depth, distance, and relative displacement. We argue that effective 3D space injection depends on two fundamental design choices: what geometric representation to inject and how to fuse it with visual features while keeping the injected geometry explicitly accessible. To this end, we introduce In3D-VLM, which directly incorporates metric 3D geometry obtained from depth sensors or metric depth estimation models. The Point-Map Tokenizer (PMT), a lightweight tokenizer pretrained through masked XYZ point-map reconstruction, converts dense observations into compact, geometrically recoverable tokens without relying on a large geometry backbone. Geometry-Preserving Fusion (GPF) then maps these geometry tokens through a fixed orthonormal basis into a dedicated subspace of the LLM embedding space and adds them to the spatially aligned RGB visual tokens. The orthonormal mapping expands the geometry tokens into the LLM space without deforming their learned geometric structure. Geometry-leak regularization further reduces semantic overlap with the reserved subspace, encouraging an explicit linear geometry readout from the fused tokens. We further introduce MD2Bench to evaluate metric depth and distance reasoning from provided 3D geometry under visual-point and text-name grounding protocols. In3D-VLM substantially outperforms existing VLMs on MDBench and achieves state-of-the-art performance on established public spatial-reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Metric Depth, Vision Language Model, Spatial Understand, 3DVLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_tzUMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates how to effectively feed given 3D coordinates into vision-language models. The input is an XYZ point map aligned pixel-by-pixel with the image. PMT performs pretraining by masking intermediate features and reconstructing coordinates, compressing the point map into a more compact representation; GPF reserves a set of directions for geometric features, adds them to image features, and through a training penalty reduces image-feature occupation of these directions, thereby supporting approximate geometric feature readback at the fusion entry point without increasing the visual input sequence length.

The second stage freezes the vision encoder, PMT, and orthogonal basis, and updates the two projectors and the language model. Point maps in the experiments come from MoGe-2 estimation. The authors additionally construct MD2Bench, which generates both inputs and answers from the same estimated geometry, posing questions via visual markers and text names to evaluate depth, camera distance, and inter-object distance.

This paper’s main advance lies in the learned point-map compression and constrained fusion method. The asterisked versions in the tables compare injection mechanisms under a shared framework. Overall, I found it an interesting work.

Critical strengths

(1) Same-condition experiments support the substantive value of the added geometric scheme.Under the same 1B framework, the overall MD2 of 68.9 also exceeds adapted SD-VLM (64.0) and N3D-VLM (66.4).

(2) The representation and fusion designs have direct verification, and the evaluation addresses a practical confound. Table 3 shows PMT achieves lower coordinate reconstruction error (0.166 m) than average pooling (0.201 m) and bilinear downscaling (0.239 m). Table 4a shows PMT+GPF improves over PMT+Add on MD2 and CV-Bench, though SPAR drops slightly. Table 4b confirms the leakage penalty reduces image-feature occupancy in geometric directions from 50.3% to 15.6% and readback error from 0.55 to 0.16. MD2 uses same-source geometry for inputs and scoring, reducing source-inconsistency artifacts.

Critical weaknesses

The following reservations affect module interpretation and evaluation reproducibility

(1) The configuration correspondence between the two ablation paths in Table 4 needs clarification.

Table 4a's "PMT+Add" and Table 4b's "+PMT, Geo. projector" report different scores (e.g., CV 85.8 vs. 83.6; SPAR 69.4 vs. 66.0), but the text does not clarify whether they share projection dimensions, fusion positions, or training setup. The two paths may use different configurations, so differing numbers alone do not prove a contradiction.

(2) MD2 evaluation details and scope of gains need clarification.

Same-source inputs and answers do not rule out RGB-based prediction; absolute MRA should not be read as "geometry utilization rate." For text-name inter-object distance, the paper's 1B/4B scores (40.7/38.9) trail adapted N3D-VLM (42.3), though visual-marker scores lead clearly. This localized shortfall does not negate overall gains but limits the scope. Data splits across training stages and MD2 scoring rules (tolerance tiers, numerical parsing) also need author confirmation.

Questions for authors
  1. Please clarify the configuration correspondence between Table 4a's PE+Add and PMT+Add rows and Table 4b's geometric projector row.Do the two PMT baselines use the same projection dimensions, fusion position, training, and evaluation settings, and do they come from the same run? If they differ, please specify the differences; if identical, please explain the score discrepancy. A configuration clarification would suffice to address this reporting question, with no presumption that retraining is required. The answer will affect my interpretation of per-module gains; further re-evaluation would only be needed if the clarification reveals substantive issues such as the controlled comparisons being invalidated.

  2. Please provide the training/evaluation splits and operational details for MD2 target selection and scoring.Key areas include image/scene exclusion relationships across stages, the SPAR training/test split, unique reference resolution for text-based questions, and handling of invalid points and responses. If error analysis is available, please indicate whether the localized shortfall in the text-name distance task is related to target or reference-point ambiguity. Providing the existing splits and operational rules would constitute an initial response; further re-evaluation of affected samples would only be needed if substantive supervision overlap or target ambiguity is discovered. The answer will help delineate the scope of the results

  3. Please supplement with a comparison to OpenSGG-VL, explaining the key differences in design philosophy and implementation between this paper's GPF—which "reserves directions for geometric features and suppresses image interference"—and OpenSGG-VL's approach of "preserving 2D feature directions and injecting orthogonal 3D residuals," to clarify the novel contribution of this paper's fusion method.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tzUM

Thank you for the constructive review. Shared questions are answered in the general response (GR).

W1 / Q1 (Table 4a vs. 4b). The rows are separate runs with identical data, steps, seed, fusion position, and evaluation. They differ only in the components below.

RowGeometryProjectorQ + leakCVSPARMD²Bench
PE + Add (4a)PE––56.364.464.7
PMT + Add (4a)PMT––85.869.466.6
+ PMT, Geo. projector (4b)PMTlinear (k = 1024)–83.666.068.4
In3D-VLM (4b, full GPF)PMTlinear (k = 512)✓89.769.068.9

The learnable projector strengthens geometry but adds interference with RGB semantics, and the remaining rows of Table 4b are designed to resolve this conflict. We clarified these settings in Sec. 5.3.

W2 (same-source targets). To separate geometry readout from same-source agreement, we swapped the input and target sources between MoGe-2 and sensor depth (GR-Q1, Table 10). Scores peak whenever the input and the targets share a source, and sensor input with sensor targets scores as high as the MoGe-2 setting (68.6 vs. 68.9). A model estimating geometry from RGB would not depend on which map is supplied, so the answers come from the injected geometry. Zeroing the map further drops MD²Bench to 25.7, below the RGB-only model (60.1). We agree that absolute MRA should not be read as a geometry utilization rate, and these tests measure geometry use directly. The text-name shortfall is addressed in Q2.

Q2 (splits, target selection, and scoring).

  • Splits: See GR-Q5.
  • Target selection: Targets come from the official SUN RGB-D polygon annotations. An object is eligible only if its name is unique in the image and not "unknown", its polygon centroid lies inside both the image and the polygon, the polygon covers at least 100 pixels, and the depth at the marker pixel is valid. Each image contributes four distinct eligible objects (fixed seed), one each for depth and object–camera distance and two for object–object distance, which must be at least 5 cm apart in 3D. Images without enough eligible objects are skipped, so MD²Bench contains no invalid points or zero-valued answers.
  • Scoring: MRA (Eq. 14) with the SPAR-Bench tolerance set. The first number in the response is parsed as the answer, and a response without a number receives zero credit.
  • Text-name distance: The shortfall comes from text grounding, not geometry readout. A grounding error on either object invalidates the answer, so 34.6% of text-name object–object answers receive zero credit, against 2.9% under point prompts. Among models sharing the same base VLM, object–object scores differ clearly under point prompts (72.3, 76.1, 81.3 / 81.8) but converge under text prompts (38.0–42.3), within the 95% CI (±2.4). Once the targets are specified, In3D-VLM reads the geometry most accurately.

Q3 (OpenSGG-VL). OpenSGG-VL's Orthogonal Residual Fusion (ORF) and GPF both use orthogonality, but for opposite ends. ORF protects the 2D semantic direction for open-vocabulary classification, while GPF reserves a subspace so that dense metric geometry can be read back from the VLM's visual tokens.

  • Task and input: ORF fuses one object-level 2D CLIP feature with one 3D point-cloud feature, both CLIP-aligned, and matches the result against category prompts. Metric quantities such as centroids, bounding boxes, and distances reach its LLM as text. GPF fuses dense per-patch metric geometry with visual tokens in the LLM embedding space.
  • Role of orthogonality: ORF adds the 3D component orthogonal to the 2D feature, normalized to unit length and scaled by a confidence gate, so the 3D signal is a secondary correction whose magnitude is discarded. GPF injects geometry at full strength through a basis Q (Z_fin = Q·Z_geo + Z_sem).
  • Readout: ORF's orthogonal direction changes with each sample and the fused vector is re-normalized, so the 3D component cannot be recovered from it. GPF's fixed orthonormal Q preserves distances (Eq. 9) and gives a linear readout (Qᵀ·Z_fin = Z_geo + Qᵀ·Z_sem), and the leak term lowers the readout error from 0.55 to 0.16 (Table 4b).
  • Training: ORF is a parameter-free fusion applied at inference. GPF is trained end-to-end with the leak term while Q stays fixed.

We hope these responses address your concerns. The Table 4 comparisons are controlled, and once the target objects are specified, In3D-VLM reads the supplied geometry more accurately than the other injection strategies.

Thanks for clarifying the ablation settings and MD2 evaluation protocol. The new geometry perturbation, rescaling, and sensor-depth tests strengthen the evidence that the model uses the supplied geometry. These additions address most of my concerns, and I maintain my score.

Reviewer_r7NfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how to inject metric 3D geometry into a VLM so it stays readable after fusion. In3D-VLM back-projects estimated metric depth (MoGe-2) into an XYZ point map and encodes it with a 52M Point-Map Tokenizer (PMT). Geometry-Preserving Fusion (GPF) maps these tokens through a fixed orthonormal basis Q into a reserved k = 512 subspace, adds them to the RGB tokens, and uses a leak penalty to keep RGB semantics out of that subspace. The authors also build MD2Bench, whose answers come from the same XYZ map the model sees. On InternVL3.5-1B and 4B, the method reports the best listed scores on CV-Bench, SPAR-Bench and SP-Bench-SI.

Critical strengths

I appreciate that the paper runs controls most spatial-VLM papers skip: an RGB-only SFT with the same base, 1.4M data and Stage-2 settings, plus SD-VLM and N3D-VLM injection re-implemented on that base (Table 2). That's what convinces me the geometry input does help: over RGB-only SFT it adds +3.1 CV, +5.2 SPAR and +13.2 MD2-VP (Table 4a), and In3D-1B beats SDVLM* in all six MD2Bench cells. I also like that the fusion is cheap (one bias-free projector, a fixed QR basis, no extra tokens), though MoGe-2's cost isn't counted.

Critical weaknesses

My main concern is that the SOTA claim mostly reflects the SFT data. In Table 4a (p.9), RGB-only SFT takes the 1B base from 67.2 to 86.6 on CV-Bench and from 32.1 to 63.8 on SPAR, so about 86% of the full model's gain (19.4 of 22.5, 31.7 of 36.9) comes with no geometry at all. On CV-Bench that row alone would beat every open-source and spatial VLM in Table 1 (best 86.2), yet it isn't in Table 1, there's no RGB-only number for SP-Bench-SI (the largest claimed lead, +6.6) or for 4B, and p.8 still credits "explicit geometry integration". Add SPAR-7M at 45.5% of Stage 2 (Table 6) with no stated scene split from the benchmarks, and I honestly don't think Table 1 says much about the architecture yet.

I'm also not convinced GPF, the "faithful" part, is isolated. In Table 4b the orthonormal basis moves MD2/CV/SPAR by -0.5/+0.8/+0.6, and the leak term's +1.0/+5.3/+2.4 lands mostly on CV-Bench, which p.9 ties to "visual-semantic representations". Correct me if I'm wrong, but L_leak (Eq. 11) depends only on the fixed Q and Z_sem = P_vis(E_vis(I)), so its gradient only reaches the RGB projector. I wonder if that CV gain is just RGB-side regularization. The Table 4a PMT/Add row and the Table 4b "+ PMT, Geo. projector" row also look like the same setting but disagree (CV 85.8 vs 83.6, SPAR 69.4 vs 66.0), and against the 4a row GPF actually lowers SPAR by 0.4. With one seed (A.2.1) and k, fusion position and the leak term all picked on the test benchmarks (Table 4b, Table 8), I can't tell that spread apart from the claimed effects.

Questions for authors
  1. Could you add RGB-only SFT rows to Table 1 (including SP-Bench-SI and 4B) and confirm the benchmark scenes are disjoint from training? If the geometry margin holds, my first concern mostly goes away.
  2. I'm curious what RGB-only SFT with the same fixed Q and leak term gets on CV-Bench. If it recovers most of the +5.3, the leak term is regularizing RGB, not protecting geometry.
  3. What differs between the two no-GPF rows in Tables 4a and 4b? A couple more seeds on the GPF rows would help.
  4. What does MD2Bench give with the XYZ map zeroed, shuffled or rescaled at inference? RGB-only SFT already gets 60.1 vs 68.9, so I think that's the most direct test of "faithful".
  5. How is geometry supplied for multi-view and video inputs, given the method is defined for one image and one XYZ map? A per-task SPAR breakdown would show whether the multi-view tasks gain at all.
  6. I am curious that if the authors compared with All-Angles Bench (Yeh et al., AAAI 2026)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer r7Nf

Thank you for the detailed review. Shared questions are answered in the general response (GR).

W1 / Q1 (SFT rows, scene overlap). We added RGB-only SFT rows to Table 1 for 1B and 4B, including SP-Bench-SI (GR-Q2). We agree that the training data explains much of the lead over prior work, and Sec. 5.2 now separates the two. In3D-VLM still improves over the same-data baseline on every benchmark at both scales, most clearly on MD²Bench (+8.8 / +8.1, GR-Q2). For overlap, see GR-Q5.

W2 / Q2 (RGB-only SFT with the same Q and leak term). We ran this control (Appendix A.8.1, Table 11).

GeometryQ + leakCVSP-SISPARMD²Bench
✗✗86.670.463.860.1
✗✓89.970.863.660.6
✓✓89.771.669.068.9

You are right that the CV-Bench gain comes from regularizing the semantic features, and we corrected Sec. 5.3. This is a side effect. The purpose of the leak term is to clear the reserved subspace for geometry, and in this control the leak loss drops by 85% during training (0.200 → 0.029, Fig. 8). Clearing it alone does not help geometry-heavy tasks (SPAR −0.2, MD²Bench +0.5). They improve only once geometry is injected into it (+5.4, +8.3).

Q3 (two no-GPF rows, seeds). The two rows share all settings except the geometry projector. PMT + Add (Table 4a) adds the frozen PMT tokens directly, while Table 4b first applies a learnable linear projector (k = D = 1024). The projector strengthens geometry (MD²Bench 66.6 → 68.4) but adds interference with RGB semantics (CV 85.8 → 83.6, SPAR 69.4 → 66.0). The remaining rows of Table 4b resolve this conflict, raising SPAR from 66.0 to 69.0. We acknowledge that k, the fusion position, and the leak term were chosen on these benchmarks with a single seed, and we could not complete more seeds within the rebuttal period. The same setting is used unchanged at 4B, where the gains over the same-data baseline hold (GR-Q2).

Q4 (geometry faithfulness). In our direct tests (GR-Q1), zeroing, rescaling, or adding noise to the input XYZ map changed the output accordingly. Since the goal is to perturb the geometry with the RGB input fixed, we used noise in place of the shuffle variant.

Q5 (multi-view and video inputs). In3D-VLM improves the overall score in both settings, mainly on metric tasks. Each image is paired with its own XYZ map and fused independently, and the fused tokens are fed to the LLM in order, as in the base InternVL. SPAR-Bench per-task results against the same-data RGB-only baseline are below.

TaskSingle-view (RGB → Ours)Multi-view (RGB → Ours)
depth_prediction_oc79.5 → 86.173.9 → 81.5
depth_prediction_oo34.3 → 45.529.1 → 43.1
distance_prediction_oc70.7 → 86.173.2 → 82.1
distance_prediction_oo63.0 → 73.553.9 → 68.8
obj_spatial_relation_oc–81.3 → 83.0
obj_spatial_relation_oo81.6 → 81.678.0 → 76.2
spatial_imagination_oc65.3 → 64.064.2 → 63.1
spatial_imagination_oo54.6 → 55.072.3 → 70.0
distance_infer_center_oo89.4 → 87.976.1 → 69.6
position_matching–60.1 → 58.5
camera_motion_infer–76.3 → 75.8
view_change_infer–0.0 → 28.8
Overall67.3 → 72.561.5 → 66.7

Overall is the unweighted mean of the listed tasks. Over all 20 tasks, the score rises from 63.8 to 69.0. On view_change_infer, the RGB-only baseline receives no credit under the official evaluation code. Excluding this task, the multi-view score still rises from 67.1 to 70.2.

Q6 (All-Angles Bench).

ModelAvg.Attr.Cam. PoseCount.Manip.Rel. Dir.Rel. Dist.
VG-LLM33.756.716.526.330.026.934.2
AoTD36.841.532.427.937.626.745.5
VLM-3R40.356.122.739.435.930.945.6
CoF47.875.735.841.438.738.749.0
SpaceR49.772.851.146.241.238.149.4
In3D-VLM-1B38.550.129.039.037.425.043.3
In3D-VLM-4B43.264.238.140.241.424.445.5

In3D-VLM-4B achieves the best Manipulation score and outperforms VG-LLM, AoTD, and VLM-3R on average, but remains below CoF and SpaceR. All-Angles Bench evaluates cross-view consistency in human-activity scenes without metric questions, which lies outside our target setting. MoGe-2 also estimates each view's point map independently, so scale and intrinsics may be inconsistent across views. Extending GPF to consistent multi-view geometry is future work.

We hope these responses address your concerns. While part of the margin over prior work comes from our training data, the same-data comparisons show consistent gains in metric spatial reasoning that RGB-only training cannot provide.

Thank you for the response and the new controls that I suggested. The same-data RGB-only rows, the leak-term control and the per-task SPAR breakdown address my main concerns, and I appreciate the corrected Sec. 5.3. I think Iwill raise my score accordingly at this point.

Thank you for the constructive discussion and for reconsidering your score. Your suggested controls clearly strengthened the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: New Controls on Geometry Faithfulness, Baselines, and Overlap

General response

We thank all reviewers for their careful feedback. We have uploaded a revised PDF with revisions marked in blue. For brevity, only the key numbers are summarized here.

Q1. Does the model actually read the injected geometry?

Yes. We ran three inference-time tests on In3D-VLM-1B (new Appendix A.7). In every test, the output changed in accordance with the altered geometry.

Test 1: Perturbing the XYZ map (Appendix A.7.1, Table 9).

Input XYZCV 2DCV 3DSPAR Num.MD²Bench
Normal86.792.666.268.9
Noise σ = 0.586.6 (−0.1)91.8 (−0.8)64.7 (−1.5)66.9 (−2.0)
Noise σ = 1.086.7 (0.0)91.3 (−1.3)61.8 (−4.4)60.1 (−8.8)
Zero84.3 (−2.4)78.8 (−13.8)26.4 (−39.8)25.7 (−43.2)

Zeroing drops MD²Bench to 25.7, below the RGB-only model (60.1), and degradation grows with noise on metric and 3D tasks, while CV-Bench 2D barely changes. Metric and 3D reasoning therefore rely on the injected geometry rather than RGB.

Test 2: Rescaling the XYZ map by α (Appendix A.7.2, Fig. 7).

When the input is scaled by α ∈ {0.25, 0.5, 2, 4}, the numeric answers on SPAR-Bench and MD²Bench scale near-linearly, close to y = αx. The model therefore computes its answers from the supplied coordinates rather than from learned size priors.

Test 3: Swapping the geometry source (Appendix A.7.3, Table 10).

Input \ AnswerMoGe-2Sensor
MoGe-268.965.2
Sensor60.968.6

Scores peak when the input and the answers share a source, and sensor depth scores as high as MoGe-2 (68.6 vs. 68.9). The model therefore reads whichever map it is given and is not tied to the MoGe-2 geometry used in training.

Q2. Do the gains hold against a same-data RGB-only baseline?

Yes. Table 1 now includes RGB-only SFT baselines trained with the same data and settings (Sec. 5.2). MD²Bench scores are added below.

ModelCVSP-SISPARMD²Bench
InternVL3.5-1B-SFT86.670.463.860.1
In3D-VLM-1B89.771.669.068.9
InternVL3.5-4B-SFT87.276.270.962.5
In3D-VLM-4B89.376.871.870.6

In3D-VLM improves over the same-data baseline on every benchmark at both scales. The largest margins are on MD²Bench (+8.8 at 1B, +8.1 at 4B), which isolates metric reasoning over the supplied geometry.

Q3. How much overhead does geometry injection add?

Geometry injection adds only a small overhead to the base VLM (new Appendix A.9).

ModelParams (M)GFLOPsTTFT (ms)
InternVL3.5-1B1060.91026.853.6
In3D-VLM-1B (given depth)1115.21102.960.6
In3D-VLM-1B + MoGe-21441.42230.394.5
In3D-VLM-1B + VGGT2273.13537.4135.6

Given depth, In3D-VLM adds 5.1% parameters, 7.4% GFLOPs, and 7.0 ms of TTFT (Table 13). With depth estimation, MoGe-2 is 30% faster than VGGT. PMT and GPF together take under 5% of TTFT (Table 14).

Q4. Is orthonormality itself necessary?

Not for general benchmarks, but for metric readout. We replaced Q with the non-orthonormal G spanning the same subspace (G = QR) under the same leak term (Appendix A.8.2, Table 12).

Q (orthonormal)G (non-orthonormal)
Semantic occupancy (%) ↓15.620.6
Readback error, pseudo-inverse ↓0.1550.264
CV / SP-SI / SPAR89.7 / 71.6 / 69.090.3 / 71.0 / 67.5
MD²Bench68.965.8

Without orthonormality, more semantic energy leaks into the geometry subspace, readback degrades, and metric reasoning drops (MD²Bench −3.1, SPAR −1.5).

Q5. Is there train–test overlap?

Our training data has no overlap with SPAR-Bench, CV-Bench-3D, or MD²Bench, and small overlaps with SP-Bench-SI and CV-Bench-2D.

  • SPAR-Bench: No overlap. We use the official SPAR-7M-RGBD training split and verified by image matching that no benchmark image appears in our training data. SPAR-7M-RGBD-Custom relabels the same split.
  • MD²Bench: No overlap. Many SUN RGB-D images appear in public corpora such as SpatialCorpus, so we sampled the 1,000 images only from those absent from our training data and confirmed this by content matching.
  • SP-Bench-SI: 2 of 170 scenes (23 of 1,009 questions, 2.3%) appear in our training data through the public SPAR-7M training split, which includes a few ScanNet validation scenes. Only one frame is shared, and the training questions differ from the benchmark questions.
  • CV-Bench: No overlap in CV-Bench-3D. In CV-Bench-2D, 146 of 1,438 questions (10.2%, Count 85 and Relation 61) use COCO/ADE20K validation images that are part of the public InternSpatial training set.

Both overlaps stem from standard datasets (COCO, ADE20K, ScanNet) reused across public training sets and benchmarks. Since our RGB-only SFT baseline is trained on the same data, the overlaps affect both models in the same way.

AuthorsOpenReview ↗

Response to Reviewer sYv6

Thank you for the careful review. Shared questions are answered in the general response (GR).

W1 / Q3 (RGB-only baseline, VSI-Bench). Table 1 now includes RGB-only SFT baselines trained with the same data and settings, so the effect of geometry can be read separately from that of the data. Against this baseline, In3D-VLM still improves every Table 1 benchmark at both scales, with the largest gains on metric reasoning (MD²Bench +8.8 at 1B, +8.1 at 4B, GR-Q2).

ModelCVSP-SISPARMD²BenchVSI
InternVL3.5-1B-SFT86.670.463.860.155.1
In3D-VLM-1B89.771.669.068.952.1
InternVL3.5-4B-SFT87.276.270.962.560.3
In3D-VLM-4B89.376.871.870.661.8

VSI-Bench evaluates scene-level reasoning that integrates information across a whole scene video, such as counting, room size, route planning, and appearance order. In3D-VLM instead targets metric reasoning from a per-frame point map, and these maps are not aligned across frames. On VSI-Bench, In3D-VLM-4B improves over its baseline (+1.5), while the 1B model falls below it (−3.0). We attribute this mixed result to the unaligned per-frame maps and regard VSI-Bench as outside our target setting. We did not run ReVSI.

W1 (comparison with N3D-VLM, text-name shortfall). Under the same base VLM and data, In3D-VLM-1B outperforms the adapted N3D-VLM on MD²Bench (68.9 vs. 66.4). Text-name object–object distance is a more challenging task, since the model must ground two objects from text before any geometry can be used. The three methods sharing our base VLM (SD-VLM*, N3D-VLM*, In3D-VLM) fall within 38.0–42.3 here, while they separate clearly under point prompts (72.3, 76.1, 81.3). This task therefore reflects text grounding rather than geometry readout.

W2 (orthonormality). Orthonormality is what preserves metric readout. We replaced Q with a fixed non-orthonormal G spanning the same subspace, under the same leak term (Table 12).

Q (orthonormal)G (non-orthonormal)
Semantic occupancy (%) ↓15.620.6
Readback error, pseudo-inverse ↓0.1550.264
CV / SP-SI / SPAR89.7 / 71.6 / 69.090.3 / 71.0 / 67.5
MD²Bench68.965.8

Without orthonormality, more semantic energy leaks into the geometry subspace, readback degrades, and metric reasoning drops (MD²Bench −3.1, SPAR −1.5). The leak term also regularizes RGB features, which keeps general understanding intact while geometry is injected. Without it, injecting geometry lowers CV-Bench to 84.4, and with it CV-Bench reaches 89.7 (Tables 4b and 11). See Appendix A.8.2.

W2 / Q2 (blanking geometry, geometry faithfulness). In3D-VLM reads the supplied geometry rather than estimating it from RGB (GR-Q1). The purpose of this test is to perturb the geometry while leaving the semantic tokens unchanged, so we zeroed the XYZ map at the input in place of token-level blanking. The RGB pathway, and thus Z_sem, is identical in both cases. MD²Bench then collapses to 25.7, far below the RGB-only model trained without geometry (60.1), and CV-Bench and SPAR fall to 81.5 and 47.7, while CV-Bench 2D changes by only 2.4 points. The output also degrades with noise strength and scales near-linearly when the map is rescaled. A model relying on RGB would show none of these effects. See Appendix A.7.

W2 (readout vs. estimation). The answers follow the supplied map, not an RGB-based estimate (GR-Q1). As suggested, we paired the test with sensor-depth targets and swapped the input and target sources between MoGe-2 and sensor depth. Scores peak when the sources match, and sensor input with sensor targets scores 68.6, as high as the MoGe-2 setting (68.9). A model estimating geometry from RGB would not depend on which map is supplied. See Appendix A.7.3.

Q1 (evaluation inputs). Each model receives geometry only in the form its design accepts.

  • RGB-only (Table 4): Image only, with no pathway for an XYZ map.
  • SD-VLM: Depth Anything V2 depth maps, per its official implementation.
  • SpaceR, SpatialStack: RGB only. SpatialStack derives geometry internally with VGGT.
  • SD-VLM*, N3D-VLM*: The same MoGe-2 XYZ map, base VLM, and data as In3D-VLM, with only the fusion replaced. These rows give the controlled comparison for the fusion method.

Q4 (MD²Bench images, precision). MD²Bench has no overlap with our training data, and 0.1 cm is the intended training precision. We sampled the 1,000 images only from SUN RGB-D images absent from our training data and confirmed this by content matching (GR-Q5). SPAR-7M-RGBD-Custom relabels the original 0.1 m targets at 0.1 cm for finer-grained metric supervision, while MD²Bench is evaluated at 0.1 m.

We hope these responses address your concerns. The new controls show that In3D-VLM reads the supplied geometry and that orthonormality is what preserves it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.