- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates the benefits of 3D language understanding for novel view generation. It adopts a pretrained unified model with specialized generation and understanding transformers with few modifications to jointly learn 3D language understanding and novel view generation. It achieves SOTA performance on novel view generation and reveals the benefits of 3D language understanding in enhancing generation capability and improving cross-view correspondences.
Critical strengths
The paper asks and tries to answer an interesting question: whether 3D language understanding can benefit novel view generation. The work is well-motivated and well-defined. The authors do a good job of explaining their methodology and sharing their findings with good writing. The setup to achieve Finding 1 is particularly interesting, and the finding itself is informative. They achieve superior performance in novel view generation among baselines.
Critical weaknesses
[Contributions and Novelties]
• (Major) The introduction of the paper lacks clear statements on claimed contributions of the work, making it difficult to assess the novelty. Moreover, the differentiation from prior methods needs more work. The related works section has been moved to appendix. This disrupts the flow of the paper and makes the research gap difficult to identify. This is understandable given space constraints but could only be acceptable if intro partly covered the comparison with prior works and provided sufficient differentiation. The related works section further needs to cite and discuss an important and relevant prior work named UniUGG [1].
[Findings and Experiments]
• (Major) I don't quite understand the significance of “Finding 2”. The paper demonstrates that joint training with 3D understanding improves generation as “Finding 3”. Isn’t Finding 2 a natural cause of Finding 3? Why is Finding 2 important, and how does it elevate our understanding of the explored relationship beyond performance improvement?
• (Minor) Presented qualitative results need improvement as they don't establish Ask4NV's superiority clearly. Including error maps or enlarged crops can help highlighting the superiority of Ask4NV.
• (Minor) Several quality metrics (e.g. ATE, CIDEr) need to be defined in the main paper before being used.
[1] Xu, Yueming, et al. "Uniugg: Unified 3d understanding and generation via geometric-semantic encoding." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
I invite the authros to focus on addressing the major points above in their rebuttal and consider minor points for future works or further improvements. At the moment, I lean towards rejection. However, I look forward to rebuttal discussion period and will be happy to reconsider my vote. I thank authors for the effort they put into presenting their interesting work.
5 comments
Response to Reviewer o5Cx (2/2)
W2: I don't quite understand the significance of “Finding 2”. The paper demonstrates that joint training with 3D understanding improves generation as “Finding 3”. Isn’t Finding 2 a natural cause of Finding 3? Why is Finding 2 important, and how does it elevate our understanding of the explored relationship beyond performance improvement?
We acknowledge that Finding 2 is a natural cause of Finding 3 and does not go far beyond the performance improvement. Our intent with Findings 1 and 2 was to analyze how the model's internal visual representations change under joint training with 3D language understanding. However, Finding 2 compares generated and ground-truth views in the feature space of an external encoder, and thus does not directly reflect this intent. To examine the internal representations directly, we additionally conduct linear depth probing on the VAE-token features of Ask4NV. We train a linear probe on frozen layer-26 features, the layer with the strongest visual correspondence in Finding 1, on the indoor benchmark NYUv2 [1] and the outdoor benchmark KITTI [2].
Table R1. Linear depth probing experiments
| NYUv2 | KITTI | |||
|---|---|---|---|---|
| AbsRel | AbsRel | |||
| Lance-FT | 0.118 | 0.871 | 0.152 | 0.799 |
| Lance-FT + ViT | 0.108 | 0.895 | 0.124 | 0.851 |
| Ask4NV | 0.093 | 0.933 | 0.101 | 0.902 |
Table R1 shows that 3D language supervision leads the VAE-token features to encode more spatial information, as measured by linear depth probing. Lance-FT + ViT and Ask4NV differ only in joint training with 3D language understanding, and this training reduces AbsRel by 14% on NYUv2 (-0.015) and 19% on KITTI (-0.023), with consistent gains in . This gives a representation-level account of how understanding benefits generation, beyond the performance gain itself. In the revised paper, we have replaced Finding 2 with this depth probing analysis and moved the feature-similarity results (Table 2) to the appendix as an additional generation metric.
W3 Qualitative results. Following your suggestion, we have updated the qualitative results in the revised paper with enlarged crops, which make the differences between Ask4NV and the baselines clearer.
W4 Metric definitions. We add all metrics with their full names and a brief description before their first use in the main paper (Sec. 6.1 and 6.3), with detailed definitions of the geometric metrics in Sec. A.4.
[1] Silberman et al., “Indoor segmentation and support inference from rgbd images.” ECCV 2012.
[2] Geiger et al., “Vision meets robotics: The kitti dataset.” 2013
Official Comment by Authors
Thank you again for your thoughtful feedback. As the discussion period is coming to a close, we would be happy to address any remaining questions or concerns. Please let us know if there is anything further we can clarify.
Response to Reviewer o5Cx (1/2)
Thank you for your constructive review and valuable suggestions, and for highlighting the setup and insight behind Finding 1. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly.
Revised introduction. Following your suggestion, we revise the introduction to clarify our contributions and the research gap. The introduction now explicitly states our research question of whether high-level semantic and spatial supervision from 3D language understanding can improve novel-view generation, and contrasts our work with prior unified 3D understanding and generation models (UniUGG and Omni-View), which do not aim to improve novel-view generation itself. We also add an explicit list of contributions at the end of the introduction, and move the related work section to the main paper (Sec. 2).
Comparison with UniUGG. Thank you for pointing out UniUGG, which unifies 3D language understanding and 3D scene generation by pretraining a geometric-semantic vision encoder. While UniUGG can produce target-view images by projecting its generated point cloud, it differs from Ask4NV in two key respects:
UniUGG is an important and relevant work on unifying 3D generation and 3D language understanding. We now cite and discuss it in both the introduction (Sec. 1) and the related work (Sec. 2) of the revised paper.