- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates how to effectively feed given 3D coordinates into vision-language models. The input is an XYZ point map aligned pixel-by-pixel with the image. PMT performs pretraining by masking intermediate features and reconstructing coordinates, compressing the point map into a more compact representation; GPF reserves a set of directions for geometric features, adds them to image features, and through a training penalty reduces image-feature occupation of these directions, thereby supporting approximate geometric feature readback at the fusion entry point without increasing the visual input sequence length.
The second stage freezes the vision encoder, PMT, and orthogonal basis, and updates the two projectors and the language model. Point maps in the experiments come from MoGe-2 estimation. The authors additionally construct MD2Bench, which generates both inputs and answers from the same estimated geometry, posing questions via visual markers and text names to evaluate depth, camera distance, and inter-object distance.
This paper’s main advance lies in the learned point-map compression and constrained fusion method. The asterisked versions in the tables compare injection mechanisms under a shared framework. Overall, I found it an interesting work.
Critical strengths
(1) Same-condition experiments support the substantive value of the added geometric scheme.Under the same 1B framework, the overall MD2 of 68.9 also exceeds adapted SD-VLM (64.0) and N3D-VLM (66.4).
(2) The representation and fusion designs have direct verification, and the evaluation addresses a practical confound. Table 3 shows PMT achieves lower coordinate reconstruction error (0.166 m) than average pooling (0.201 m) and bilinear downscaling (0.239 m). Table 4a shows PMT+GPF improves over PMT+Add on MD2 and CV-Bench, though SPAR drops slightly. Table 4b confirms the leakage penalty reduces image-feature occupancy in geometric directions from 50.3% to 15.6% and readback error from 0.55 to 0.16. MD2 uses same-source geometry for inputs and scoring, reducing source-inconsistency artifacts.
Critical weaknesses
The following reservations affect module interpretation and evaluation reproducibility
(1) The configuration correspondence between the two ablation paths in Table 4 needs clarification.
Table 4a's "PMT+Add" and Table 4b's "+PMT, Geo. projector" report different scores (e.g., CV 85.8 vs. 83.6; SPAR 69.4 vs. 66.0), but the text does not clarify whether they share projection dimensions, fusion positions, or training setup. The two paths may use different configurations, so differing numbers alone do not prove a contradiction.
(2) MD2 evaluation details and scope of gains need clarification.
Same-source inputs and answers do not rule out RGB-based prediction; absolute MRA should not be read as "geometry utilization rate." For text-name inter-object distance, the paper's 1B/4B scores (40.7/38.9) trail adapted N3D-VLM (42.3), though visual-marker scores lead clearly. This localized shortfall does not negate overall gains but limits the scope. Data splits across training stages and MD2 scoring rules (tolerance tiers, numerical parsing) also need author confirmation.
Questions for authors
-
Please clarify the configuration correspondence between Table 4a's PE+Add and PMT+Add rows and Table 4b's geometric projector row.Do the two PMT baselines use the same projection dimensions, fusion position, training, and evaluation settings, and do they come from the same run? If they differ, please specify the differences; if identical, please explain the score discrepancy. A configuration clarification would suffice to address this reporting question, with no presumption that retraining is required. The answer will affect my interpretation of per-module gains; further re-evaluation would only be needed if the clarification reveals substantive issues such as the controlled comparisons being invalidated.
-
Please provide the training/evaluation splits and operational details for MD2 target selection and scoring.Key areas include image/scene exclusion relationships across stages, the SPAR training/test split, unique reference resolution for text-based questions, and handling of invalid points and responses. If error analysis is available, please indicate whether the localized shortfall in the text-name distance task is related to target or reference-point ambiguity. Providing the existing splits and operational rules would constitute an initial response; further re-evaluation of affected samples would only be needed if substantive supervision overlap or target ambiguity is discovered. The answer will help delineate the scope of the results
-
Please supplement with a comparison to OpenSGG-VL, explaining the key differences in design philosophy and implementation between this paper's GPF—which "reserves directions for geometric features and suppresses image interference"—and OpenSGG-VL's approach of "preserving 2D feature directions and injecting orthogonal 3D residuals," to clarify the novel contribution of this paper's fusion method.
2 comments
Thanks for clarifying the ablation settings and MD2 evaluation protocol. The new geometry perturbation, rescaling, and sensor-depth tests strengthen the evidence that the model uses the supplied geometry. These additions address most of my concerns, and I maintain my score.
Response to Reviewer tzUM
Thank you for the constructive review. Shared questions are answered in the general response (GR).
W1 / Q1 (Table 4a vs. 4b). The rows are separate runs with identical data, steps, seed, fusion position, and evaluation. They differ only in the components below.
The learnable projector strengthens geometry but adds interference with RGB semantics, and the remaining rows of Table 4b are designed to resolve this conflict. We clarified these settings in Sec. 5.3.
W2 (same-source targets). To separate geometry readout from same-source agreement, we swapped the input and target sources between MoGe-2 and sensor depth (GR-Q1, Table 10). Scores peak whenever the input and the targets share a source, and sensor input with sensor targets scores as high as the MoGe-2 setting (68.6 vs. 68.9). A model estimating geometry from RGB would not depend on which map is supplied, so the answers come from the injected geometry. Zeroing the map further drops MD²Bench to 25.7, below the RGB-only model (60.1). We agree that absolute MRA should not be read as a geometry utilization rate, and these tests measure geometry use directly. The text-name shortfall is addressed in Q2.
Q2 (splits, target selection, and scoring).
Q3 (OpenSGG-VL). OpenSGG-VL's Orthogonal Residual Fusion (ORF) and GPF both use orthogonality, but for opposite ends. ORF protects the 2D semantic direction for open-vocabulary classification, while GPF reserves a subspace so that dense metric geometry can be read back from the VLM's visual tokens.
Z_fin = Q·Z_geo + Z_sem).Qᵀ·Z_fin = Z_geo + Qᵀ·Z_sem), and the leak term lowers the readout error from 0.55 to 0.16 (Table 4b).We hope these responses address your concerns. The Table 4 comparisons are controlled, and once the target objects are specified, In3D-VLM reads the supplied geometry more accurately than the other injection strategies.