- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how vision-language models (VLMs) can better utilize features from pretrained geometry models to improve spatial question answering from multiple views. The authors observe that adding geometry features under standard spatial QA supervision yields limited gains on high-level spatial reasoning benchmarks. They attribute this to a training signal that allows the model to rely on visual appearance and language priors without sufficiently encouraging the integration of geometric and semantic information.
The paper therefore proposes novel-view semantic rendering as an auxiliary task. Given context images and a target camera token, the model predicts a 16 × 16 semantic grid for the target view. Visual features are fused with VGGT geometry features through cross-attention with a gated residual connection before being passed to the language model. The auxiliary task uses 256 learnable spatial query tokens and a separate two-layer MLP classification head, shared across grid locations, to predict all grid-cell labels in a single forward pass.
Controlled experiments using InternVL3.5-4B and Qwen3-VL-4B show that adding this auxiliary task improves average scores on VSI-Bench, ReVSI, and 3D-Point-QA over the corresponding geometry-augmented baselines.
Critical strengths
-
The auxiliary task is well motivated by the target capability, and its implementation addresses a potential training shortcut. Novel-view semantic prediction is a reasonable candidate objective for encouraging the combination of geometric structure and object semantics. Using spatial query tokens with a separate classification head avoids feeding preceding ground-truth semantic labels into the decoder, thereby removing the specific teacher-forcing shortcut discussed in Section 3.2.
-
The gains show some consistency across two backbones and multiple benchmarks. The controlled comparisons in Tables 3(a) and 3(b) provide stronger evidence for the contribution of the auxiliary task than rankings across complete systems with different training configurations. Improvements occur across all three benchmark averages on both backbones, although they are not uniform across subtasks. The point-prompt design of 3D-Point-QA also provides a useful way to evaluate geometric reasoning with reduced ambiguity from text-based object grounding.
Critical weaknesses
-
The target-information leakage analysis does not test the target camera token itself, leaving uncertainty in the interpretation of the auxiliary task. Section 3.2 obtains this token by jointly processing the target RGB frame and context frames with frozen VGGT, then discarding the target-view patch tokens. Appendix A.5 tests leakage through the context-view features while keeping the target camera token unchanged. It therefore does not rule out the possibility that this token carries target-specific semantic or layout information beyond camera pose. This limits the evidence for interpreting the task as strictly pose-conditioned prediction of an unobserved view and for attributing its success to the intended integration of context geometry and semantics.
-
The existing ablations do not fully isolate the contribution of novel-view prediction from additional semantic supervision and training exposure. The comparisons involving RGB prediction, geometry features, grid resolution, and alternative auxiliary objectives are informative. However, they do not fully establish how much of the improvement specifically depends on predicting semantics at a novel viewpoint, rather than additional semantic learning or other multitask training effects. Appendix A.1 reports identical training schedules across ablations, but the actual update counts and QA exposure under the different task mixtures remain insufficiently explicit.
Questions for authors
-
What evidence rules out target-specific semantic or layout information in the target camera token beyond camera pose? Appendix A.5 keeps this token unchanged and therefore addresses leakage through the context features rather than through the token itself.
-
Beyond the identical schedules reported in Appendix A.1, could you clarify the actual number of optimizer updates, the QA-to-auxiliary sampling ratio, and the number of times each QA sample is seen during training for every setting in Tables 3(a) and 3(b)? When auxiliary samples are removed, how are these quantities controlled?
Response to Reviewer 9r7u
We thank the reviewer for the constructive comments. We answer the raised questions below.
W1/Q1: "What evidence rules out target-specific semantic or layout information in the target camera token beyond camera pose? Appendix A.5 keeps this token unchanged and therefore addresses leakage through the context features rather than through the token itself."
We trained a variant (Pose-enc) in which the latent target camera token is replaced with the 9-d pose encoding (translation, rotation and field of view) decoded by VGGT's camera head, so only explicit camera parameters are passed to the model (Qwen3-VL-4B).
The VSI-Bench gain is preserved (68.4 vs. 68.2) and 3DPQA still improves over the + VGGT baseline (78.6 vs. 77.1), while ReVSI is on par with + VGGT. The benefit of novel-view semantic prediction therefore holds under strictly pose-conditioned prediction.
On 5,000 held-out samples from 490 scenes not seen in training, novel-view rendering with Qwen3-VL-4B drops from 73.7 patch accuracy / 54.6 mIoU with the camera token to 69.3 / 48.6 with Pose-enc. (The rendering numbers in the paper, 44.7 mIoU and 67.0 patch accuracy, are from InternVL3.5-4B and are not directly comparable.) Rendering from pose alone is somewhat harder, but rendering accuracy is not strictly tied to QA performance, which is our main target: Pose-enc matches the camera-token model on VSI-Bench.
Q2: "Beyond the identical schedules reported in Appendix A.1, could you clarify the actual number of optimizer updates, the QA-to-auxiliary sampling ratio, and the number of times each QA sample is seen during training for every setting in Tables 3(a) and 3(b)? When auxiliary samples are removed, how are these quantities controlled?"
All models in Tables 3(a) and 3(b) are exposed to the same QA samples: 394K, each seen once, since we train for one epoch with batch size 384 (Appendix A.1). The 100K auxiliary samples are added on top, a QA-to-auxiliary ratio of about 4:1, which gives about 1.29K optimizer updates. When the auxiliary samples are removed, we keep the same batch size, so those models take fewer optimization steps (about 1.03K); the QA exposure is unchanged.
We hope these clarifications and results address the reviewer's questions, and we are happy to answer any further questions.