acceptodds
Under review as a conference paper at ICLR 2027

Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs

Abstract

Recent works augment Vision-Language Models with geometry features from pretrained 3D models, expecting that the geometric signal will boost spatial reasoning. However, we find that simply fusing geometry features and training on standard spatial QA yields only marginal improvements on high-level multi-hop tasks. We attribute this gap to a training-signal problem: standard spatial QA can be largely answered from visual features and language priors, so the geometry pathway receives weak gradients and fails to integrate with the visual features. To encourage the model to use both features, we propose novel-view semantic rendering as an auxiliary training task that requires the model to predict the semantic layout of an unobserved viewpoint, inspired by humans' ability to mentally simulate novel viewpoints during spatial reasoning. This task encourages joint use of both pathways: geometry provides pose-dependent visibility, while vision provides semantic content. Our auxiliary task yields consistent improvements over the geometry-augmented baseline across all three benchmarks (+1.6 on VSI-Bench, +2.2 on ReVSI, +2.9 on our 3D-Point-QA dataset), and our full model surpasses prior open-source methods on ReVSI and is on par with the strongest baseline on VSI-Bench with less training data. The code and data will be openly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Reasoning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9r7uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how vision-language models (VLMs) can better utilize features from pretrained geometry models to improve spatial question answering from multiple views. The authors observe that adding geometry features under standard spatial QA supervision yields limited gains on high-level spatial reasoning benchmarks. They attribute this to a training signal that allows the model to rely on visual appearance and language priors without sufficiently encouraging the integration of geometric and semantic information.

The paper therefore proposes novel-view semantic rendering as an auxiliary task. Given context images and a target camera token, the model predicts a 16 × 16 semantic grid for the target view. Visual features are fused with VGGT geometry features through cross-attention with a gated residual connection before being passed to the language model. The auxiliary task uses 256 learnable spatial query tokens and a separate two-layer MLP classification head, shared across grid locations, to predict all grid-cell labels in a single forward pass.

Controlled experiments using InternVL3.5-4B and Qwen3-VL-4B show that adding this auxiliary task improves average scores on VSI-Bench, ReVSI, and 3D-Point-QA over the corresponding geometry-augmented baselines.

Critical strengths
  1. The auxiliary task is well motivated by the target capability, and its implementation addresses a potential training shortcut. Novel-view semantic prediction is a reasonable candidate objective for encouraging the combination of geometric structure and object semantics. Using spatial query tokens with a separate classification head avoids feeding preceding ground-truth semantic labels into the decoder, thereby removing the specific teacher-forcing shortcut discussed in Section 3.2.

  2. The gains show some consistency across two backbones and multiple benchmarks. The controlled comparisons in Tables 3(a) and 3(b) provide stronger evidence for the contribution of the auxiliary task than rankings across complete systems with different training configurations. Improvements occur across all three benchmark averages on both backbones, although they are not uniform across subtasks. The point-prompt design of 3D-Point-QA also provides a useful way to evaluate geometric reasoning with reduced ambiguity from text-based object grounding.

Critical weaknesses
  1. The target-information leakage analysis does not test the target camera token itself, leaving uncertainty in the interpretation of the auxiliary task. Section 3.2 obtains this token by jointly processing the target RGB frame and context frames with frozen VGGT, then discarding the target-view patch tokens. Appendix A.5 tests leakage through the context-view features while keeping the target camera token unchanged. It therefore does not rule out the possibility that this token carries target-specific semantic or layout information beyond camera pose. This limits the evidence for interpreting the task as strictly pose-conditioned prediction of an unobserved view and for attributing its success to the intended integration of context geometry and semantics.

  2. The existing ablations do not fully isolate the contribution of novel-view prediction from additional semantic supervision and training exposure. The comparisons involving RGB prediction, geometry features, grid resolution, and alternative auxiliary objectives are informative. However, they do not fully establish how much of the improvement specifically depends on predicting semantics at a novel viewpoint, rather than additional semantic learning or other multitask training effects. Appendix A.1 reports identical training schedules across ablations, but the actual update counts and QA exposure under the different task mixtures remain insufficiently explicit.

Questions for authors
  1. What evidence rules out target-specific semantic or layout information in the target camera token beyond camera pose? Appendix A.5 keeps this token unchanged and therefore addresses leakage through the context features rather than through the token itself.

  2. Beyond the identical schedules reported in Appendix A.1, could you clarify the actual number of optimizer updates, the QA-to-auxiliary sampling ratio, and the number of times each QA sample is seen during training for every setting in Tables 3(a) and 3(b)? When auxiliary samples are removed, how are these quantities controlled?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 9r7u

We thank the reviewer for the constructive comments. We answer the raised questions below.

W1/Q1: "What evidence rules out target-specific semantic or layout information in the target camera token beyond camera pose? Appendix A.5 keeps this token unchanged and therefore addresses leakage through the context features rather than through the token itself."

We trained a variant (Pose-enc) in which the latent target camera token is replaced with the 9-d pose encoding (translation, rotation and field of view) decoded by VGGT's camera head, so only explicit camera parameters are passed to the model (Qwen3-VL-4B).

SettingVSI-BenchReVSI3DPQA
+ VGGT (Table 3b)67.051.577.1
+ VGGT + Sem., R2R (Table 3b)68.252.580.0
+ VGGT + Sem., Pose-enc (new)68.451.678.6

The VSI-Bench gain is preserved (68.4 vs. 68.2) and 3DPQA still improves over the + VGGT baseline (78.6 vs. 77.1), while ReVSI is on par with + VGGT. The benefit of novel-view semantic prediction therefore holds under strictly pose-conditioned prediction.

On 5,000 held-out samples from 490 scenes not seen in training, novel-view rendering with Qwen3-VL-4B drops from 73.7 patch accuracy / 54.6 mIoU with the camera token to 69.3 / 48.6 with Pose-enc. (The rendering numbers in the paper, 44.7 mIoU and 67.0 patch accuracy, are from InternVL3.5-4B and are not directly comparable.) Rendering from pose alone is somewhat harder, but rendering accuracy is not strictly tied to QA performance, which is our main target: Pose-enc matches the camera-token model on VSI-Bench.

Q2: "Beyond the identical schedules reported in Appendix A.1, could you clarify the actual number of optimizer updates, the QA-to-auxiliary sampling ratio, and the number of times each QA sample is seen during training for every setting in Tables 3(a) and 3(b)? When auxiliary samples are removed, how are these quantities controlled?"

All models in Tables 3(a) and 3(b) are exposed to the same QA samples: 394K, each seen once, since we train for one epoch with batch size 384 (Appendix A.1). The 100K auxiliary samples are added on top, a QA-to-auxiliary ratio of about 4:1, which gives about 1.29K optimizer updates. When the auxiliary samples are removed, we keep the same batch size, so those models take fewer optimization steps (about 1.03K); the QA exposure is unchanged.

We hope these clarifications and results address the reviewer's questions, and we are happy to answer any further questions.

Reviewer_rZ2BOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper combines VLM visual features with frozen VGGT features and adds novel-view semantic prediction as auxiliary supervision. A target camera token conditions a coarse semantic grid; the rendering head is unused during QA inference. Experiments on two backbone families show improvements on spatial QA benchmarks and a new point-prompt dataset.

Critical strengths
  1. Improvements over the geometry-augmented baseline occur on both backbones and all three main benchmarks. This supports the practical value of the auxiliary objective.
  2. Table 3 includes useful comparisons of geometry features, semantic versus RGB targets, grid resolution, and GeoSR. These controls make the empirical contribution more convincing than the headline comparison alone.
Critical weaknesses
  1. The target camera token is extracted from the target RGB image. Appendix A.5 removes target information from context features but retains this token, leaving possible semantic information in the target query untested. This weakens the central interpretation of predicting an unobserved view; it does not establish leakage into downstream QA evaluation.
  2. An input-view semantic-prediction control is missing, so the gains may partly reflect additional semantic supervision rather than viewpoint reasoning. Exact update-budget matching and variability are also unclear, which matters for the mostly 1–2-point high-level gains.
Questions for authors
  1. Does replacing the target latent token with decoded camera parameters preserve the rendering and downstream QA gains?
  2. How does input-view semantic prediction perform under the same supervision and update budget? Were optimizer updates and QA exposure matched in Table 3?
1 comment
AuthorsOpenReview ↗

Response to Reviewer rZ2B

We thank the reviewer for the constructive comments. We ran the two experiments you asked for, using Qwen3-VL-4B and address the questions below.

  • Pose-enc. The latent target camera token is replaced with the 9-d pose encoding (translation, rotation and field of view) decoded by VGGT's camera head, so only explicit camera parameters are passed to the model.
  • Input-view. The auxiliary task predicts the semantic map of an input view instead of a novel view.
SettingVSI-BenchReVSI3DPQA
+ VGGT (Table 3b)67.051.577.1
+ VGGT + Sem., R2R (Table 3b)68.252.580.0
+ VGGT + Sem., Pose-enc (new)68.451.678.6
+ VGGT + Sem., Input-view (new)67.349.575.9

W1/Q1: "Does replacing the target latent token with decoded camera parameters preserve the rendering and downstream QA gains?"

With decoded camera parameters (Pose-enc), the VSI-Bench gain is preserved (68.4 vs. 68.2) and 3DPQA still improves over the + VGGT baseline (78.6 vs. 77.1), while ReVSI is on par with + VGGT. The benefit of novel-view semantic prediction therefore does not rely on the latent camera token. As you note, the rendering head and target token are not used at QA inference.

For rendering quality, we evaluate these Qwen3-VL-4B models on 5,000 held-out samples from 490 scenes not seen in training. Note that the rendering numbers in the paper (44.7 mIoU, 67.0 patch accuracy) are from InternVL3.5-4B, so they are not directly comparable to the Qwen numbers below.

Semantic rendering (Qwen3-VL-4B)Patch accuracymIoU
Camera token (novel view)73.754.6
Pose-enc (novel view)69.348.6
Input-view (target view among the inputs)85.872.8

Rendering from decoded camera parameters is somewhat harder, so rendering accuracy drops. However, rendering accuracy is not strictly tied to QA performance, which is our main target: Pose-enc matches the camera-token model on VSI-Bench despite the lower rendering accuracy.

W2/Q2: "How does input-view semantic prediction perform under the same supervision and update budget? Were optimizer updates and QA exposure matched in Table 3?"

Predicting the semantics of an input view (Input-view) gives essentially no gain over the + VGGT baseline on VSI-Bench (+0.3) and is lower on ReVSI and 3DPQA, while novel-view prediction improves over it. On the same held-out samples, the Input-view model reaches a much higher 85.8 patch accuracy and 72.8 mIoU, since it sees the target view, yet this does not translate into QA gains. The gains therefore come from reasoning about an unobserved viewpoint rather than from additional semantic supervision.

On the update budget: all models in Table 3 are exposed to the same QA samples (394K, each seen once in one epoch) with the same batch size of 384. The 100K auxiliary samples are added on top, so models with the auxiliary task take about 1.29K optimization steps and models without it about 1.03K.

We hope these clarifications and results address the reviewer's questions, and we are happy to answer any further questions.

Reviewer_dirFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Render2Reason (R2R), a training framework that introduces novel-view semantic rendering as an auxiliary task to improve spatial reasoning in vision-language models (VLMs). Given multi-view images and a target camera pose, the model predicts the semantic layout of an unobserved viewpoint, encouraging the integration of visual semantics and geometric features extracted by VGGT. The authors also introduce 3D-Point-QA, a benchmark designed to evaluate low-level geometric reasoning through point-based queries. Experiments on VSI-Bench, ReVSI, and 3D-Point-QA demonstrate consistent improvements over geometry-augmented baselines.

Critical strengths
  1. Novel-view semantic prediction connects camera pose, visibility, and scene semantics, providing a more direct geometric learning signal than standard spatial QA alone. The consistent improvements over geometry-augmented baselines across multiple benchmarks and two backbone models support the practical value of this training strategy.

  2. The authors investigate geometry features, semantic rendering, RGB rendering, grid resolution, and GeoSR across VSI-Bench, ReVSI, and 3D-Point-QA. These experiments provide useful insights into the conditions under which auxiliary supervision is beneficial, rather than relying solely on aggregate results from a single benchmark.

Critical weaknesses
  1. The paper attributes its improvements to more effective integration of visual and geometric information, but the evidence primarily consists of performance comparisons, modality-removal experiments, and qualitative rendering results. Removing the geometry pathway at inference introduces a distribution shift, so the resulting performance degradation does not independently establish increased reliance on geometric information during normal inference. More rigorous controlled experiments are needed to substantiate the proposed mechanism.

  2. Potential scene-level overlap between training data and evaluation benchmarks, as well as generalization to different scenes and viewpoint distributions, is not sufficiently investigated. These omissions weaken the claims regarding the method's robustness and general applicability.

  3. The experimental evaluation covers a relatively limited set of competing methods and does not establish a sufficiently comprehensive comparison against the latest relevant approaches. The proprietary-model baselines are also limited, with some comparisons relying on older models, and the paper does not systematically evaluate R2R against the strongest currently available proprietary systems under matched settings.

Questions for authors

Is there any scene-level overlap between ScanNet, ScanNet++, and ARKitScenes used for training and the evaluation benchmarks? How do the authors ensure that no scene-level leakage occurs across training, validation, and test sets?

The zero-shot results on MindCube-Tiny in the appendix show that improvements are concentrated in the Among and Around settings, whereas performance on Rotation remains below that of the original pretrained model. Does this suggest a substantial dependence on the training viewpoint distribution, and how well does the method generalize to substantially different camera configurations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dirF

We thank the reviewer for the constructive comments. We address the concerns and questions below.

W2/Q1: "Is there any scene-level overlap between ScanNet, ScanNet++, and ARKitScenes used for training and the evaluation benchmarks? How do the authors ensure that no scene-level leakage occurs across training, validation, and test sets?"

There is no scene-level overlap. All training data (the VSI-590K and VLM-3R QA pairs, our novel-view semantic samples, and the 3D-Point-QA training set) is sampled only from the training splits of ScanNet, ScanNet++ and ARKitScenes, while all evaluation questions come from the validation splits. We will state this explicitly in the paper.

W3: "The experimental evaluation covers a relatively limited set of competing methods and does not establish a sufficiently comprehensive comparison against the latest relevant approaches."

Our study focuses on how to improve an existing model with a better training objective, so the main analysis fixes the backbone, data and schedule and changes only the auxiliary task (Sec. 4). For the system-level comparison, Tables 1 and 2 already include Cambrian-P, which to our knowledge is the strongest open-source model on these benchmarks. R2R is higher on both VSI-Bench (74.4 vs. 73.7) and ReVSI (54.5 vs. 52.0), with fewer training QA samples (394K vs. 590K).

For proprietary models, Table 2 compares with GPT-5.2 and Gemini 3 Pro on ReVSI. We agree that the proprietary baselines on VSI-Bench in Table 1 are older, so we evaluated Claude Opus 5.5 on VSI-Bench with 128 uniformly sampled frames, the same number of frames our model uses.

VSI-Bench taskR2R (Qwen3-VL, 9B)Claude Opus 5.5
Object counting75.062.0
Absolute distance64.154.1
Object size78.376.5
Room size75.043.0
Relative distance80.078.2
Relative direction88.481.6
Route planning48.575.3
Appearance order85.995.5
Average74.470.8

R2R is higher on average and on six of eight tasks. Opus 5.5 is stronger on route planning and appearance order, so even the latest proprietary model is not better on every task, and neither model dominates. Note that R2R is finetuned on spatial QA while the proprietary model is evaluated zero-shot. We will add this comparison and further recent baselines to the revision.

Q2: "The zero-shot results on MindCube-Tiny in the appendix show that improvements are concentrated in the Among and Around settings, whereas performance on Rotation remains below that of the original pretrained model. Does this suggest a substantial dependence on the training viewpoint distribution, and how well does the method generalize to substantially different camera configurations?"

We agree that this reflects a dependence on the training distribution, and we believe spatial QA finetuning is the main cause rather than the rendering objective. In Table 10, the largest drop on Rotation already occurs with QA finetuning alone: the Finetuned baseline, which uses neither geometry features nor semantic rendering, falls from 37.5 to 30.5, and R2R (32.0) is above it. Rotation consists of orthogonal views from an in-place rotating camera, which is quite different from the QA samples used for finetuning. Our semantic rendering samples are extracted from continuous video frames, so the auxiliary task does not cover this configuration either and does not recover the drop. MindCube is never seen during training, and on Among and Around, whose camera configurations are closer to multi-view observation of a scene, R2R is the best of all variants (41.5 and 50.8), giving the best overall accuracy (41.9). We will discuss Rotation as a limitation in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.