acceptodds
Under review as a conference paper at ICLR 2027

Can Learning to Answer Spatial Questions Improve Novel-View Generation?

Abstract

Generating novel views from limited observations requires preserving both the appearance and spatial structure of a scene. Although recent approaches exploit fine-grained geometric cues, they do not explicitly model high-level semantic and spatial properties such as object identity, attributes, and inter-object relationships. In this work, we investigate whether 3D language understanding can provide complementary supervision for novel-view generation. We introduce Ask4NV, built on a pretrained unified multimodal model with a Mixture-of-Transformers (MoT) architecture. Our model incorporates camera geometry into both VAE and ViT representations and leverages understanding features during generation through joint self-attention with a modality-aware attention masking strategy. Experiments demonstrate strong image quality and geometric consistency in novel-view generation, alongside competitive 3D language understanding. Further analysis shows that joint training improves cross-view correspondence in VAE representations and similarity in a 3D-aware feature space.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel view generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_o5CxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates the benefits of 3D language understanding for novel view generation. It adopts a pretrained unified model with specialized generation and understanding transformers with few modifications to jointly learn 3D language understanding and novel view generation. It achieves SOTA performance on novel view generation and reveals the benefits of 3D language understanding in enhancing generation capability and improving cross-view correspondences.

Critical strengths

The paper asks and tries to answer an interesting question: whether 3D language understanding can benefit novel view generation. The work is well-motivated and well-defined. The authors do a good job of explaining their methodology and sharing their findings with good writing. The setup to achieve Finding 1 is particularly interesting, and the finding itself is informative. They achieve superior performance in novel view generation among baselines.

Critical weaknesses

[Contributions and Novelties]

• (Major) The introduction of the paper lacks clear statements on claimed contributions of the work, making it difficult to assess the novelty. Moreover, the differentiation from prior methods needs more work. The related works section has been moved to appendix. This disrupts the flow of the paper and makes the research gap difficult to identify. This is understandable given space constraints but could only be acceptable if intro partly covered the comparison with prior works and provided sufficient differentiation. The related works section further needs to cite and discuss an important and relevant prior work named UniUGG [1].

[Findings and Experiments]

• (Major) I don't quite understand the significance of “Finding 2”. The paper demonstrates that joint training with 3D understanding improves generation as “Finding 3”. Isn’t Finding 2 a natural cause of Finding 3? Why is Finding 2 important, and how does it elevate our understanding of the explored relationship beyond performance improvement?

• (Minor) Presented qualitative results need improvement as they don't establish Ask4NV's superiority clearly. Including error maps or enlarged crops can help highlighting the superiority of Ask4NV.

• (Minor) Several quality metrics (e.g. ATE, CIDEr) need to be defined in the main paper before being used.

[1] Xu, Yueming, et al. "Uniugg: Unified 3d understanding and generation via geometric-semantic encoding." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

I invite the authros to focus on addressing the major points above in their rebuttal and consider minor points for future works or further improvements. At the moment, I lean towards rejection. However, I look forward to rebuttal discussion period and will be happy to reconsider my vote. I thank authors for the effort they put into presenting their interesting work.

5 comments
AuthorsOpenReview ↗

Response to Reviewer o5Cx (1/2)

Thank you for your constructive review and valuable suggestions, and for highlighting the setup and insight behind Finding 1. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly.

W1: The introduction of the paper lacks clear statements on claimed contributions of the work, making it difficult to assess the novelty. Moreover, the differentiation from prior methods needs more work. The related works section has been moved to appendix. This disrupts the flow of the paper and makes the research gap difficult to identify. This is understandable given space constraints but could only be acceptable if intro partly covered the comparison with prior works and provided sufficient differentiation. The related works section further needs to cite and discuss an important and relevant prior work named UniUGG [1].

Revised introduction. Following your suggestion, we revise the introduction to clarify our contributions and the research gap. The introduction now explicitly states our research question of whether high-level semantic and spatial supervision from 3D language understanding can improve novel-view generation, and contrasts our work with prior unified 3D understanding and generation models (UniUGG and Omni-View), which do not aim to improve novel-view generation itself. We also add an explicit list of contributions at the end of the introduction, and move the related work section to the main paper (Sec. 2).

Comparison with UniUGG. Thank you for pointing out UniUGG, which unifies 3D language understanding and 3D scene generation by pretraining a geometric-semantic vision encoder. While UniUGG can produce target-view images by projecting its generated point cloud, it differs from Ask4NV in two key respects:

  • Research focus. UniUGG focuses on how to unify 3D language understanding and 3D scene generation, and addresses this by pretraining a geometric-semantic vision encoder. It does not examine whether learning the understanding task benefits generation. This question, how 3D language understanding can enhance novel-view generation, is the main focus of our work.
  • Task formulation. UniUGG generates a 3D scene as a point cloud from a single source image, and obtains a target image by projecting the point cloud. Ask4NV instead generates target-view images directly from multiple source views.

UniUGG is an important and relevant work on unifying 3D generation and 3D language understanding. We now cite and discuss it in both the introduction (Sec. 1) and the related work (Sec. 2) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer o5Cx (2/2)

W2: I don't quite understand the significance of “Finding 2”. The paper demonstrates that joint training with 3D understanding improves generation as “Finding 3”. Isn’t Finding 2 a natural cause of Finding 3? Why is Finding 2 important, and how does it elevate our understanding of the explored relationship beyond performance improvement?

We acknowledge that Finding 2 is a natural cause of Finding 3 and does not go far beyond the performance improvement. Our intent with Findings 1 and 2 was to analyze how the model's internal visual representations change under joint training with 3D language understanding. However, Finding 2 compares generated and ground-truth views in the feature space of an external encoder, and thus does not directly reflect this intent. To examine the internal representations directly, we additionally conduct linear depth probing on the VAE-token features of Ask4NV. We train a linear probe on frozen layer-26 features, the layer with the strongest visual correspondence in Finding 1, on the indoor benchmark NYUv2 [1] and the outdoor benchmark KITTI [2].

Table R1. Linear depth probing experiments

NYUv2KITTI
AbsRel ↓\downarrowδ<1.25↑\delta_{<1.25} \uparrowAbsRel ↓\downarrowδ<1.25↑\delta_{<1.25} \uparrow
Lance-FT0.1180.8710.1520.799
Lance-FT + ViT0.1080.8950.1240.851
Ask4NV0.0930.9330.1010.902

Table R1 shows that 3D language supervision leads the VAE-token features to encode more spatial information, as measured by linear depth probing. Lance-FT + ViT and Ask4NV differ only in joint training with 3D language understanding, and this training reduces AbsRel by 14% on NYUv2 (-0.015) and 19% on KITTI (-0.023), with consistent gains in δ<1.25\delta_{<1.25}. This gives a representation-level account of how understanding benefits generation, beyond the performance gain itself. In the revised paper, we have replaced Finding 2 with this depth probing analysis and moved the feature-similarity results (Table 2) to the appendix as an additional generation metric.

W3 Qualitative results. Following your suggestion, we have updated the qualitative results in the revised paper with enlarged crops, which make the differences between Ask4NV and the baselines clearer.

W4 Metric definitions. We add all metrics with their full names and a brief description before their first use in the main paper (Sec. 6.1 and 6.3), with detailed definitions of the geometric metrics in Sec. A.4.

[1] Silberman et al., “Indoor segmentation and support inference from rgbd images.” ECCV 2012.

[2] Geiger et al., “Vision meets robotics: The kitti dataset.” 2013

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for your thoughtful feedback. As the discussion period is coming to a close, we would be happy to address any remaining questions or concerns. Please let us know if there is anything further we can clarify.

Reviewer_vez7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces and interesting research question; whether supervision from 3D language understanding tasks can improve novel-view generation. The authors propose Ask4NV, built on the pretrained Mixture-of-Transformers model Lance-3B. It is jointly fine-tuned on flow-matching novel-view generation and on 3D question answering and visual grounding. Experiments on DL3DV and ScanNet (in-domain) and RE10k and ScanNet++ (out-of domain) report improved image generation and geometric consistency over generation-only baselines. The models 3D understanding capabilities are also competitive. The papers analysis suggests that joint training improves cross-view correspondences in the generation transformers VAE features and increases similarity to ground truth. While the research question is interesting and the experiments and baselines are well chosen, I believe that the presented evidence does not yet isolate the 3D language supervision as the main cause of the presented gains. The improvements are smallest on ScanNet datasets, where the language data "lives" (which is strange) and no variance across runs is reported.

Critical strengths
  • The paper proposes an interesting research question, namely if VQA and spatial grounding can improve novel view generation tasks. Asking whether high-level semantic supervision from 3D language tasks can complement the fine-level geometric cues that are typical for NVG is a new and interesting direction in my opinion. Also, the paper effectively uses figures (namely figure 2) to illustrate the intuition.

  • The proposed design is simple and builds on prior work in a nice, continous way.

  • The presented evaluations are broad in nature, containing four benchmarks with in-domain and out-of-domain splits, which are also clearly labeled. The paper presents 2D image metrics (PSNR, ...) and 3D metrics (ATE, ...) against a reasonable set of baselines.

  • The finding that correspondence improves in the generation transformer’s VAE features, without any direct correspondence supervision, is interesting if it holds up (see weaknesses).

  • The paper is also well written, easy to follow and the reproducibility details are all there.

Critical weaknesses

I do have some concerns regarding the experiments provided in the paper:

  1. In my opinion, the causal relationship between the claim and what the experiments actually show is not established. As I understand it, joint training of the novel-view generation and VQA/grounding does add 32 extra ScanNet frames per understanding sample, adding more visual data and more gradient signal. This means that Ask4NV does see more data during training than the baselines! It is therefore possible that the gains are coming from this additional visual data instead of the language objective. This experiment essentially needs a control, where we do joint training on the same frames with a non-spatial or non-language objective.

  2. The gains doe not match up with what I would expect. In Table 7. comparison between Lance+FT and Ask4NV show that joint training helps mainly on DL3DV (+1.06 dB PSNR) and RE10k(+1.01db), while it helps less on the ScanNet datasets. On ScanNet, the actual domain where the language data comes from, the gains are smaller. If the benefit comes truly from understanding scene semantics, I would expect the reverse pattern (better on ScanNet). .

3, The paper does not report seeds, repeated runs or confidence intervals. It makes it really difficult to whether the differences in Table 7 exceed run to run variance. Results over multiple seeds would substantially strengthen the claim.

  1. The results in Table 3 are not fully supporting the claims of the paper, as I believe the claim of best performance across multiple benchmarks has at least partially to do with training data distribution. Ask4NV uses ScanNet and DL3DV as training data, while CAT3D, CAMEO and Omni-View’s generation module were trained on Re10k only. On Re10K Ask4NV essentially ties CAMEO on PSNR. It trails CAMEO and Matrix3D on SSIM and on MEt3R. On ScanNet++, Matrix3D is better on SSIM, RPEr and MEt3R and ties on ATE. In addition, CAT3D and CAMEO are evaluated with VGGT-estimated cameras while other methods receive ground-truth cameras. This difference should be discussed as a potential for errors, or removed by running these baselines with ground-truth cameras

Minor issues.

  • In Sec. 2 (“Shared scene properties across tasks”), a sentence and duplicated (“…relies on recognizing characteristic feature that learning to interpret scene properties… These examples suggest that learning to interpret scene properties…”).
Questions for authors
  • Does the understanding loss L_und produce any gradient in the generation transformers parameters? Or doe the effects reach generation only thr;ough the understanding transformers ViT representation through joint self-attention?
  • Can we train the understanding transformer on the same 32-frame ScanNet samples with a non-spatial objective (e.g., generic captioning or shuffled answers) and report generation results? This would separate the effect of spatial-language supervision from the effect of additional visual data.
  • How much do results vary across random seeds? Reporting mean/std over several runs would be helpful.
  • Joint training shows the largest gains on DL3DV and Re10K, but only about +0.2 dB on ScanNet and ScanNet++, the domains of the language data. On ScanNet, Lance-FT + ViT is also better on ATE, RPEt and MEt3R. How do you reconcile this with the proposed mechanism?
  • How much does using VGGT-estimated cameras for CAT3D and CAMEO, versus ground-truth cameras for the other methods, affect the Table 3 comparison?
9 comments
AuthorsOpenReview ↗

Response to Reviewer vez7 (1/7)

Thank you for highlighting our research question and the correspondence analysis, and for your insightful suggestions. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly. And also we thank for catching minor. We correct the sentence in the revised paper.

W1: In my opinion, the causal relationship between the claim and what the experiments actually show is not established. As I understand it, joint training of the novel-view generation and VQA/grounding does add 32 extra ScanNet frames per understanding sample, adding more visual data and more gradient signal. This means that Ask4NV does see more data during training than the baselines! It is therefore possible that the gains are coming from this additional visual data instead of the language objective. This experiment essentially needs a control, where we do joint training on the same frames with a non-spatial or non-language objective.

Q2: Can we train the understanding transformer on the same 32-frame ScanNet samples with a non-spatial objective (e.g., generic captioning or shuffled answers) and report generation results? This would separate the effect of spatial-language supervision from the effect of additional visual data.

We first note that the understanding samples in Ask4NV are drawn from the same ScanNet scenes already used for novel-view generation training. Joint training therefore introduces no new scenes.

As suggested, we conduct two control experiments: (i) a label-shuffled control, which trains on the same 32-frame ScanNet samples with the question–answer correspondence broken, and (ii) a non-spatial auxiliary control, which replaces the 3D language data with the same number of generic video QA samples. We have added these experiments to Sec. B.2 of the revised paper.

Label-shuffled control. This control directly separates the effect of the additional visual data from that of the language objective. We randomly permute the labels of the 3D language understanding data while keeping the inputs unchanged. The shuffled-label model differs from Ask4NV only in whether the labels correctly correspond to the scenes, and both models are trained for 400 optimizer steps (728 in the main experiments).

Table R1. Same frames with an uninformative objective

MethodPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
DL3DVNone (Lance-FT+ViT)12.9470.2980.422
Shuffled labels13.5240.3020.411
Correct labels (Ask4NV)14.1430.3240.378
ScanNetNone (Lance-FT+ViT)14.1790.5400.535
Shuffled labels14.2350.5430.537
Correct labels (Ask4NV)14.8700.5590.520

Table R1 demonstrates that correct 3D language supervision benefits novel-view generation, rather than acting merely as an additional training stream. Shuffled labels yield a partial gain on DL3DV (+0.58 dB over Lance-FT + ViT) and a comparable result on ScanNet (+0.06 dB). Correct labels further improve PSNR by 0.62 dB on DL3DV and 0.64 dB on ScanNet over the shuffled baseline.

AuthorsOpenReview ↗

Response to Reviewer vez7 (2/7)

non-spatial auxiliary control. We replace all 3D language understanding data (ScanQA, SQA3D, ScanRefer, and Multi3DRefer) with generic video QA from LLaVA-Hound [1], using the public 255K release distributed with VG-LLM [2]. We use the same number of understanding data as Ask4NV, keep all other settings identical, and train for 400 optimizer steps.

Table R2. Equal-budget comparison with a non-spatial auxiliary task.

DatasetObjectivePSNR ↑SSIM ↑LPIPS ↓
DL3DVNone (Lance-FT+ViT)12.9470.2980.422
Generic QA13.6240.3070.397
Ask4NV14.1430.3240.378
ScanNetNone (Lance-FT+ViT)14.1790.5400.535
Generic QA13.7170.5380.543
Ask4NV14.8700.5590.520

Table R2 shows that the gain of Ask4NV comes from the 3D semantic and spatial content of the supervision, not from the auxiliary stream alone. Ask4NV outperforms generic QA on all metrics on both datasets (+0.52 dB PSNR on DL3DV and +1.15 dB on ScanNet). Generic QA improves over Lance-FT + ViT on DL3DV (+0.68 dB) but falls below it on ScanNet (−0.46 dB).

Cross-view correspondence analysis. We further measure cross-view correspondence in the intermediate VAE features, following our analysis in Sec. 5, to examine how the model's internal representations change under each control (Table R3).

Table R3. Cross-view correspondence (PCK@1 / @2 / @4) at 400 optimizer steps, measured at attention-layer 26

ModelScanNetScanNet++
Lance-FT22.356 / 33.109 / 50.17221.347 / 31.560 / 49.916
Lance-FT + ViT35.354 / 50.560 / 67.97829.704 / 42.569 / 61.378
Generic QA36.867 / 50.853 / 67.24635.355 / 48.034 / 63.784
Shuffled labels30.475 / 44.121 / 61.73029.732 / 41.881 / 58.647
Correct labels (Ask4NV)39.290 / 53.740 / 69.37137.608 / 49.848 / 65.184

Ask4NV achieves the highest correspondence on both datasets. Generic QA yields partial improvements over Lance-FT + ViT, while shuffled labels do not improve it and even reduce it on ScanNet. Since geometric correspondence is closely tied to novel-view synthesis quality [3], this indicates that the benefit depends on the correct correspondence between language and 3D scene content.

AuthorsOpenReview ↗

Response to Reviewer vez7 (3/7)

W2: The gains doe not match up with what I would expect. In Table 7. comparison between Lance+FT and Ask4NV show that joint training helps mainly on DL3DV (+1.06 dB PSNR) and RE10k(+1.01db), while it helps less on the ScanNet datasets. On ScanNet, the actual domain where the language data comes from, the gains are smaller. If the benefit comes truly from understanding scene semantics, I would expect the reverse pattern (better on ScanNet). Q4: Joint training shows the largest gains on DL3DV and Re10K, but only about +0.2 dB on ScanNet and ScanNet++, the domains of the language data. On ScanNet, Lance-FT + ViT is also better on ATE, RPEt and MEt3R. How do you reconcile this with the proposed mechanism?

We thank the reviewer for the careful analysis of Table 7 (Table 9 of revised paper). We conduct detailed analyses of the results on ScanNet (Table R4, R5, and R6), which we add to Sec. B.3 of the revised paper.

Smaller PSNR gains on ScanNet. We measure the viewpoint change as the rotation between adjacent views among the sampled source and target views, normalized by the field of view of each camera. During training, we sample source and target views with diverse viewpoint changes across as many scenes as possible, resulting in a median viewpoint change of 0.15 on DL3DV and 0.36 on ScanNet. For evaluation, we instead sample frames uniformly over each scene, matching the frame spacing used for spatial understanding. This setting is applied identically to all methods and yields a median of 0.18 on DL3DV but 0.46 on ScanNet, thus the ScanNet views share much less overlap.

We further evaluate with diverse frame intervals and group the evaluation samples by viewpoint change (Table R4). We apply the same analysis to ScanNet++, which is not part of our training data, as our 3D language annotations come only from ScanNet.

Table R4. PSNR under different viewpoint changes. Rotation / FoV denotes the rotation between adjacent views divided by the horizontal field of view of each dataset.

DatasetRotation / FoVLance-FT + ViTAsk4NV
DL3DV0.18 (median)13.87814.939
ScanNet0.46 (median)15.57415.747
< 0.2217.96918.927
0.22–0.4517.53017.918
0.45–0.6715.56815.736
ScanNet++0.36 (median)15.03015.242
< 0.2218.05419.349
0.22–0.4515.45615.797
0.45–0.6714.31814.364

On both ScanNet and ScanNet++, the gain of Ask4NV over Lance-FT + ViT grows as the views share more overlap. For viewpoint changes below 0.22, the gain reaches +0.96 dB on ScanNet and +1.30 dB on ScanNet++, close to the gain on DL3DV (+1.06 dB). This suggests that the smaller gain on ScanNet stems from the larger viewpoint changes in our evaluation setting, rather than from the ScanNet domain itself.

Geometric metrics on ScanNet. As in Table R4, we report the geometric metrics on ScanNet by viewpoint change.

Table R5. Geometric metrics on ScanNet under different viewpoint changes (Lance-FT + ViT / Ask4NV).

Rotation / FoVATE ↓RPE_r ↓RPE_t ↓MEt3R ↓
< 0.220.035 / 0.0331.58° / 1.59°0.074 / 0.0710.325 / 0.334
0.22–0.450.059 / 0.0552.97° / 3.26°0.115 / 0.1120.383 / 0.393
0.45–0.670.104 / 0.1475.43° / 6.21°0.232 / 0.3710.437 / 0.475
> 0.670.350 / 0.33144.25° / 40.61°0.540 / 0.5760.550 / 0.574

For viewpoint changes below 0.45, the two models perform comparably. The gap appears mainly for viewpoint changes above 0.45, which are less frequent during training.

Reviewer_Zr4dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether 3D language understanding can provide complementary supervision for novel-view generation (NVG). Ask4NV fine-tunes Lance-Video-3B, a Mixture-of-Transformers unified multimodal model, with three modifications: injecting Plücker-ray camera geometry into the VAE projector and ViT tokens; coupling the generation and understanding experts via joint self-attention with a modality-aware mask; and making attention between clean source and noisy target VAE tokens bidirectional. Training combines a flow-matching NVG objective with 3D question-answering and visual-grounding losses. Experiments evaluate in-domain and out-of-domain datasets on image quality and camera-pose errors against several baselines, with additional analyses and ablations.

Critical strengths
  1. The control ladder makes the central claim testable: Lance-FT, Lance-FT+ViT and Ask4NV differ only in ViT conditioning and the presence of 3D language objectives, with the understanding side reported alongside. This isolated comparison is the main reason the paper is worth reviewing.

  2. The analysis is mechanism-oriented rather than leaderboard-oriented: it directly measures cross-view correspondence in VAE token space and patch similarity in a 3D-aware feature space, and reports an honest negative result for adding VAE conditioning to the understanding branch. This is the part most likely to survive into a revised version.

Critical weaknesses
  1. The primary claim is not identified: Ask4NV differs from its ViT-conditioned ablation in both the 3D language losses and an extra training stream, with no equal-budget control using a non-3D auxiliary task, no QA-only/grounding-only decomposition, and no label-shuffled control. The observed gains are therefore equally consistent with “3D language supervision” and with “any extra auxiliary task or tokens acting as a regularization.”

  2. On the one dataset whose supervision domain matches the evaluation domain, the geometric claim fails: Ask4NV is worse than the ViT-conditioned ablation on most geometric metrics, and its PSNR gain is smallest there and largest on a dataset with no understanding annotations. The paper reports these deltas but never discusses this pattern, while still claiming the best ATE across all datasets.

Questions for authors
  1. Please clarify the exact per-step sample composition of the three configurations: how are the 3:1 packs filled for the baselines, do all configurations see the same total tokens and FLOPs, and how many understanding samples correspond to 728×64 micro-steps? In what unit is “one epoch over the 3D language understanding data” counted?

  2. How do you reconcile Table 7 with the claim that joint training improves geometric consistency? On ScanNet, the ViT-conditioned ablation is better on ATE, RPE_t and MEt3R, and the PSNR margin is much smaller than on DL3DV. Is this a scene-count effect, over-fitting, or evidence that the gain is not specific to 3D language supervision?

  3. Which rows of Table 3 are your own runs? Please add per-row camera source, resolution, number of views and sampling steps/NFE, and report a controlled comparison at identical camera source, resolution and NFE.

  4. What is the isolated contribution of the Plücker-ray conditioning? Please add no-ray/VAE-ray-only/ViT-ray-only ablations, and state how this conditioning differs from the camera conditioning already present in Lance-Video-3B.

  5. How will the method scale beyond ScanNet-style data, given that the largest generation corpus (DL3DV) has no 3D language annotations? Does the method require per-scene 3D language annotation at training scale, and what is the intended path for unlabelled scenes?

11 comments
AuthorsOpenReview ↗

Response to Reviewer Zr4d (1/8)

Thank you for your positive assessment of our controlled comparison and analysis, and for your valuable feedback. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly.

W1: The primary claim is not identified: Ask4NV differs from its ViT-conditioned ablation in both the 3D language losses and an extra training stream, with no equal-budget control using a non-3D auxiliary task, no QA-only/grounding-only decomposition, and no label-shuffled control. The observed gains are therefore equally consistent with “3D language supervision” and with “any extra auxiliary task or tokens acting as a regularization.”

Following your suggestion, we conduct all three experiments: an equal-budget control with (i) a non-3D auxiliary task, (ii) a QA-only / grounding-only decomposition, and (iii) a label-shuffled control. Due to the limited time of the rebuttal period, all controls use a reduced training budget, which we specify for each experiment below. We add these results, together with a correspondence analysis, to Appendix B.2 of the revised paper.

Equal-budget control with a non-3D auxiliary task. We replace all 3D language understanding data (ScanQA, SQA3D, ScanRefer, and Multi3DRefer) with generic video QA from LLaVA-Hound [1], using the public 255K release distributed with VG-LLM [2]. We use the same number of understanding data as Ask4NV, keep all other settings identical, and train for 400 optimizer steps (728 in the main experiments).

Table R1. Equal-budget comparison with a non-3D auxiliary task.

MethodPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
DL3DVLance-FT+ViT12.9470.2980.422
Generic QA13.6240.3070.397
Ask4NV14.1430.3240.378
ScanNetLance-FT+ViT14.1790.5400.535
Generic QA13.7170.5380.543
Ask4NV14.8700.5590.520

Table R1 shows that the gain of Ask4NV comes from the 3D semantic and spatial content of the supervision, not from the auxiliary stream alone. Ask4NV outperforms generic video QA on all metrics on both datasets (+0.52 dB PSNR on DL3DV and +1.15 dB on ScanNet). Generic video QA improves over Lance-FT + ViT on DL3DV (+0.68 dB) but falls below it on ScanNet (−0.46 dB).

QA-only / grounding-only decomposition. We decompose our 3D language understanding data into question answering (SQA3D and ScanQA) and visual grounding (ScanRefer and Multi3DRefer), and train with each type alone to examine its contribution to novel-view generation. Since the grounding data contain only 80K samples, we compare all baselines at 300 optimizer steps. This corresponds to 0.72 epochs of QA for QA-only, 0.95 epochs of grounding for grounding-only, and 0.41 epochs over the combined QA and grounding data. We additionally report Lance-FT + ViT at the same budget as a reference without any 3D language supervision.

Table R2. Decomposition of 3D language supervision.

DatasetQAGroundingPSNR ↑SSIM ↑LPIPS ↓
DL3DV✗✗12.7830.2890.466
✓✗13.6610.3120.402
✗✓13.0310.2910.453
✓✓12.9270.2850.451
ScanNet✗✗13.1640.5010.566
✓✗13.9400.5230.523
✗✓13.4960.5140.552
✓✓13.5360.5100.546

Both QA-only and grounding-only supervision improve over Lance-FT + ViT (Table R2), indicating that each type contributes to novel-view generation. QA-only yields the largest gain (+0.88 dB on DL3DV and +0.78 dB on ScanNet), and grounding-only also improves (+0.25 dB and +0.33 dB). Ask4NV falls below QA-only at this budget, likely because it sees only 0.41 epochs over the combined data, compared with 0.72 epochs over the QA data for QA-only. The benefit of the combined supervision grows with training, and at 400 optimizer steps Ask4NV already improves over Lance-FT + ViT by 1.20 dB on DL3DV and 0.69 dB on ScanNet (Table R1).

AuthorsOpenReview ↗

Response to Reviewer Zr4d (2/8)

Label-shuffled control. We examine whether the improvement of Ask4NV comes from the correct correspondence between 3D scenes and language labels, or merely from exposure to the same 3D scene inputs and auxiliary losses. We randomly permute the labels of the 3D language understanding data while keeping the inputs unchanged, which breaks only this correspondence. The shuffled-label model and Ask4NV use the same loss formulation and training budget (400 optimizer steps).

Table R3. Label-shuffled control.

MethodPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
DL3DVNone (Lance-FT+ViT)12.9470.2980.422
Shuffled labels13.5240.3020.411
Correct labels (Ask4NV)14.1430.3240.378
ScanNetNone (Lance-FT+ViT)14.1790.5400.535
Shuffled labels14.2350.5430.537
Correct labels (Ask4NV)14.8700.5590.520

Table R3 demonstrates that correct 3D language supervision benefits novel-view generation, rather than acting merely as an additional training stream. Shuffled labels yield a partial gain on DL3DV (+0.58 dB over Lance-FT + ViT) and a comparable result on ScanNet (+0.06 dB), similar to the generic-QA control in Table R1. Correct labels further improve PSNR by 0.62 dB on DL3DV and 0.64 dB on ScanNet over the shuffled baseline.

Cross-view correspondence analysis. We further measure cross-view correspondence in the intermediate VAE features, following our analysis in Sec. 5, to examine how the model's internal representations change under each control (Table R4).

Table R4. Cross-view correspondence (PCK@1 / @2 / @4) at 400 optimizer steps, measured at attention-layer 26

ModelScanNetScanNet++
Lance-FT22.356 / 33.109 / 50.17221.347 / 31.560 / 49.916
Lance-FT + ViT35.354 / 50.560 / 67.97829.704 / 42.569 / 61.378
Generic QA36.867 / 50.853 / 67.24635.355 / 48.034 / 63.784
Shuffled labels30.475 / 44.121 / 61.73029.732 / 41.881 / 58.647
Correct labels (Ask4NV)39.290 / 53.740 / 69.37137.608 / 49.848 / 65.184

Ask4NV achieves the highest correspondence on both datasets. Generic QA yields partial improvements over Lance-FT + ViT, while shuffled labels do not improve it and even reduce it on ScanNet. Since geometric correspondence is closely tied to novel-view synthesis quality [3], this indicates that the benefit depends on the correct correspondence between language and 3D scene content.

AuthorsOpenReview ↗

Response to Reviewer Zr4d (3/8)

W2: On the one dataset whose supervision domain matches the evaluation domain, the geometric claim fails: Ask4NV is worse than the ViT-conditioned ablation on most geometric metrics, and its PSNR gain is smallest there and largest on a dataset with no understanding annotations. The paper reports these deltas but never discusses this pattern, while still claiming the best ATE across all datasets.

Q2: How do you reconcile Table 7 with the claim that joint training improves geometric consistency? On ScanNet, the ViT-conditioned ablation is better on ATE, RPE_t and MEt3R, and the PSNR margin is much smaller than on DL3DV. Is this a scene-count effect, over-fitting, or evidence that the gain is not specific to 3D language supervision?

We thank the reviewer for the careful analysis of Table 7 (Table 9 of revised paper). Motivated by this comment, we conduct detailed analyses of the results on ScanNet (Table R5, R6, and R7), which we have added to Appendix B.3 of the revised paper.

Smaller PSNR gains on ScanNet. We measure the viewpoint change as the rotation between adjacent views among the sampled source and target views, normalized by the field of view of each camera. During training, we sample source and target views with diverse viewpoint changes across as many scenes as possible, resulting in a median viewpoint change of 0.15 on DL3DV and 0.36 on ScanNet. For evaluation, we instead sample frames uniformly over each scene, matching the frame spacing used for spatial understanding. This setting is applied identically to all methods and yields a median of 0.18 on DL3DV but 0.46 on ScanNet, thus the ScanNet views share much less overlap.

We further evaluate with diverse frame intervals and group the evaluation samples by viewpoint change (Table R5). We apply the same analysis to ScanNet++, which is not part of our training data, as our 3D language annotations come only from ScanNet.

Table R5. PSNR under different viewpoint changes. Rotation / FoV denotes the rotation between adjacent views divided by the horizontal field of view of each dataset.

DatasetRotation / FoVLance-FT + ViTAsk4NV
DL3DV0.18 (median)13.87814.939
ScanNet0.46 (median)15.57415.747
< 0.2217.96918.927
0.22–0.4517.53017.918
0.45–0.6715.56815.736
ScanNet++0.36 (median)15.03015.242
< 0.2218.05419.349
0.22–0.4515.45615.797
0.45–0.6714.31814.364

On both ScanNet and ScanNet++, the gain of Ask4NV over Lance-FT + ViT grows as the views share more overlap. For viewpoint changes below 0.22, the gain reaches +0.96 dB on ScanNet and +1.30 dB on ScanNet++, close to the gain on DL3DV (+1.06 dB). This suggests that the smaller gain on ScanNet stems from the larger viewpoint changes in our evaluation setting, rather than from the ScanNet domain itself.

Geometric metrics on ScanNet. As in Table R5, we report the geometric metrics on ScanNet by viewpoint change (Table R6).

Table R6. Geometric metrics on ScanNet under different viewpoint changes (Lance-FT + ViT / Ask4NV).

Rotation / FoVATE ↓RPE_r ↓RPE_t ↓MEt3R ↓
< 0.220.035 / 0.0331.58° / 1.59°0.074 / 0.0710.325 / 0.334
0.22–0.450.059 / 0.0552.97° / 3.26°0.115 / 0.1120.383 / 0.393
0.45–0.670.104 / 0.1475.43° / 6.21°0.232 / 0.3710.437 / 0.475
> 0.670.350 / 0.33144.25° / 40.61°0.540 / 0.5760.550 / 0.574

For viewpoint changes below 0.45, the two models perform comparably. The gap appears mainly for viewpoint changes above 0.45, which are less frequent during training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.