- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
I would like to first clarify that no generative AI was used for review. The submitted paper has not been exposed to LLM analysis from my side. The current review has also not been cross-checked via LLM. This is in hope to keep the reviewing process as confidential as possible, while reflecting the opinion of the reviewer as much as possible.
The paper introduces VISA, a supervision framework that uses an offline VLM to improve the predictions of a 3D occupancy model. Before training, Qwen3-VL-8b receives crops of representative instances from the scene. It then audits each crop and outputs representative data such as class, confidence score. Each of these audits are attached to the same instance over the temporal timesteps. During training, the audited crops are used to provide further supervision to the occupancy prediction model through taxonomy, attribute and graph auditing scores. At inference, the occupancy network is queried without all the VLM scaffolding, essentially not inducing any cost.
Critical strengths
- VISA bridges the world of vision-language models and occupancy prediction, which are two trendy and impactful domains of computer vision.
- The method developed in this paper is sound and the performance gain from VISA clearly matches intuition.
- Moreover, these performance gains are non negligible.
- The idea of using a soft distribution blending thanks to VLM auditing is quite interesting. As far as I know this is an original approach (other VLM approaches supervise in a different way). The taxonomy loss that stems from it effectively repurposes a classic cross-entropy loss.
- The method does not limit itself to simply blending the VLM distributions to the dataset distribution, but also uses other interesting auditing cues such as confusion classes, object attributes (although it severely lacks description, this is interesting), enriching the distillation process.
- Since the auditing is performed offline, VISA induces no cost during during training.
Critical weaknesses
All that follows are, in my opinion, critical weaknesses.
Paper structure and clarity
- Clarity and conciseness. The paper is overall hard to read, uses a lot of unnecessary jargon and could be heavily simplified (I develop some examples in the following items). I would invite the authors to heavily polish the manuscript and simplify many of the parts, probably by rewriting them manually and clearly. This would heavily improve the impact of the manuscript.
- Better explanation of the motivation experiment. From L. 159 until L. 177, it is not clear that this loss is used as a pretext experiment to motivate the introduction of VISA. It is essential to make it clear. Moreover, L. 167-171 feel like an enumeration of different experiments rather than a point being made or a discussed result. The conclusion of these experiments only comes at L. 175, and everything written before is hard to understand. The figure 2 that accompanies this paragraph is also complicated to read "paired text cosine measures projected 3D feature–caption similarity" is supposed to be a single unit of text, but is not clear. This would improve readability and reproduction.
- Too much reference to the supplementary for the main method. Appendix A should be moved into the main paper such that readers can understand the manuscript on its own. This might come with shrinking some parts, yet, the problem formulation should remain clear. The "Offline VLM audits and Object Voxels" (L. 717) and "Learning objective" (L. 739) paragraphs are especially important and should not be relegated to the appendices. Supplementary B, especially the crop extraction and view selection, as well as the training losses should also be brought back to the main manuscript. Such reference is understandable for minor experimental details or additional experiments, but should not be used for main components of the method. Because of that, L.156 is confusing "In this section" actually refers to section 3 but sounds like it is referring to the appendix. This results in cases where variables introduced in the supplementary materials are used in the main manuscript without context (e.g. ). This makes reading especially hard. Similarly, it is not exactly clear what the attribute vector is (L. 231). Clarifying what it represents is important since it is used to created the class-to-attribute matrix (L. 231), which is involved in the attribute loss term via .
- Hallucinated reference. The RenderOcc paper (M. Pan et al., 2024) is reported as CPVR, but has been published in ICRA 2024. This and other synthetic-feeling sentences hint towards a paper that has been improperly proofread and AI-generated. This plays against the quality of the work and I would urge authors to please recheck all their experimental results, references and writings. This will improve the quality of the manuscript and strengthen the submission.
Method clarification
- Specification of the visual attributes . Referenced multiple times in the method section and defined in supplementary A, L. 722 and 724, it is still unclear what the "predefined visual attributes" are. It would seem they are binary answerable string such as "two_wheeler" (L. 233) but it is never explicitly stated. In that case, how are they made or selected? Deducting from the shape of , there should be a total of attributes for each object. I also suppose them to be shared? But at no moment they are explicitly defined. I would kindly ask authors to clarify this point. The "Blind VLM audit prompt" (L. 762) which is used to prompt the VLM for auditing itself seems ambiguous: "binary visual attributes useful for 3D semantic occupancy" (L. 770) is the only information given to the model, although it is supposed to output these predefined attributes. Could authors precisely clarify this?
Experimental suggestions
- VLM ablation. in a similar spirit, ablations with at least one other VLM would be appreciated. Using Molmo2-7b would be interesting since it is different from Qwen. If credits allow, using the ChatGPT API would be an interesting experiment too, allowing to assess the difference between open- and closed-source models. This would once again help framing the performance of the method better.
- Strengthening the benchmark results. While the results of the main benchmark are appealing, it would be interesting to evaluate the approach at least with other widely used outdoor datasets, i.e., Waymo and KITTI. The concern that I have is that VISA is hard to adapt to other datasets. Could authors report results on these additional datasets? This would heavily strengthen the impact and scope of the method. It is also expected from deep learning approaches that they generalize well.
From here on, I detail minor weaknesses.
Minor weaknesses
- Reference format. Citations should be in parentheses. Please refer to the official ICLR format.
- Unclear figure 1. In figure 1, it is unclear how the two rows interact. Adding an arrow that goes from the second row to the first row might clarify it. It is also unclear if the "audit" solid green box is also exactly the "offline VLM auditor". Carefully finetuning that figure would improve the understanding of the method.
- Referring supplementary materials. Please indicate that the reference is in the supplementary material when referring to content that is located there (e.g. Eq. 13, L. 188).
- List of all closed-set classes. It would be appreciated if the authors could list all the closed-set classes used when querying the VLM (L. 188).
- Clarification on cropped images. Why are images qualified as "sparse" (L. 194), although they are semantically dense since cropped around the object of interest?
- Better function introduction. ⊮ should be introduced before, or right after use (L. 215). I think it is supposed to be an indicator function ?
- Some sentences feel extremely synthetic and are not useful for understanding. Please, consider removing the following sentences "not treated as calibrated probabilities" (L.210). There are also a lot of sentences in the form "it is not X, but rather Y", in which one of the clause is not necessary and simply confuse reader more than anything (L. 234, 251, 409, 727, I will not list them all). This feels very synthetic in my opinion.
- Loss coefficients. Please specify the loss coefficients for the VISA loss in the main manuscript instead of the supplementary materials, so readers can understand what drives learning.
- Table 3 location. The gap between table 3c and section 4.4 looks strange. Perhaps moving the three tables to the top of the page might give render better.
- Closed-set agreement statistics. The closed-set agreement between the VLM and the training split of nuScenes is reported at 87.7% in table 4. This is very interesting. I would be curious to know if the 12.3% disagreement is more of a VLM failure or a nuScene annotation issue. I think investigating statistics like average loss on the disagreeing samples vs. average loss on the agreeing samples might be interesting to better understand how VISA drives and improves the learning.
- Main figure mislabeled. In figure 1, the VLM auditor box annotates the object with "track" instead of "truck".
Questions for authors
Please, refer to the weaknesses section.
In the current form, the weaknesses of the paper outweigh the strengths of the paper. Nonetheless, the core idea developed in VISA is interesting and worth investigation. I would invite authors to clearly assess the points raised in the weaknesses sections.
Specifically, I would like to see results of VISA on other outdoor datasets (KITTI, Waymo), showing that the method could be applied in more diverse scenarios. The ablation experiments would be more than appreciated, but not a priority in comparison to the KITTI and Waymo results. I would also like the manuscript to be heavily clarified.
Most of my comments are not experimental requests, therefore I hope that the authors can address all my concerns during the rebuttal period.
If all these issues are solved, I will consider raising my score to borderline accept or accept.
3 comments
- Additional outdoor benchmark: SemanticKITTI.
Thank you for suggesting additional outdoor evaluations. We further evaluated VISA on SemanticKITTI, using two monocular semantic scene completion architectures, MonoScene and ET-Former.
| Method | Occ. IoU ↑ | mIoU ↑ |
|---|---|---|
| MonoScene | 34.16 | 11.08 |
| + VISA | 34.30 (+0.14) | 11.25 (+0.17, +1.53%) |
| ET-Former | 51.49 | 16.30 |
| + VISA | 51.55 (+0.06) | 16.42 (+0.12, +0.74%) |
Values in parentheses indicate absolute improvements in percentage points and, for mIoU, relative improvements. Unlike the six-camera nuScenes setting, these experiments use only a single RGB image, providing a complementary evaluation under more limited visual observations. VISA improves semantic mIoU on both architectures, with relative gains of 1.53% and 0.74%, respectively. Together with our nuScenes and NYUv2 results, these experiments extend VISA to three datasets across indoor and outdoor environments, further supporting its generalizability across different visual input settings and occupancy architectures.
5.Response to Minor Comments (8–18)
Thank you for these detailed suggestions. We will address the presentation, notation, and formatting issues in the revised manuscript.
- References and notation (8, 10, 13): We will correct the citation formatting, explicitly indicate references to supplementary equations, and define the indicator function at its first use.
- Figures and layout (9, 16, 18): We will revise Fig. 1 to clarify the connection between offline VLM auditing and training-time supervision, distinguish the auditor from its output audit, and correct "track" to "truck". We will also improve the placement of Table 3.
- Terminology and writing (12, 14): We will replace the ambiguous use of "sparse images" with "object-centric crops" and simplify repetitive or unnecessarily complicated sentences throughout the manuscript.
- Closed-set classes (11): We will explicitly list the ten nuScenes object classes used for VLM auditing: barrier, bicycle, bus, car, construction vehicle, motorcycle, pedestrian, traffic cone, trailer, and truck.
- Loss coefficients (15): We will include the VISA loss coefficients in the main text for clarity.
Analysis of VLM–annotation disagreements (17).
We appreciate this suggestion. To better understand the 12.3% disagreement between VLM predictions and nuScenes annotations, we further analyzed the characteristics of agreeing and disagreeing audits.
| Metric | Agree | Disagree |
|---|---|---|
| Mean VLM class confidence | 0.697 | 0.507 |
| Mean visual reliability | 0.652 | 0.497 |
| Mean object distance | 20.30 m | 27.78 m |
| Median crop area | 18,416 px² | 9,285 px² |
| Poor visibility | 1.60% | 18.75% |
Disagreeing audits tend to have lower confidence and visual reliability, greater object distances, and substantially smaller crops. Notably, poor visibility is nearly 12 times more frequent** among disagreements. These findings suggest that limited visual evidence is an important factor associated with VLM–annotation disagreements, further motivating VISA's reliability-aware filtering and weighting.
While these statistics cannot definitively distinguish VLM recognition errors from annotation ambiguities, they provide additional insight into when disagreements arise and why treating all VLM predictions as equally reliable supervision would be undesirable. We will incorporate this analysis into the revised manuscript.
Reply assessment and final score.
Thank you for your replies.
The authors clarified and dissipated some of my concerns. I especially found the VLM ablation analysis particularly interesting, however I still have doubts related to the generalization of the approach.
Although the auditing information are quite comprehensive and should be able to improve the performance of the model on KITTI, gains are extremely marginal, this seems to imply that the method is not necessarily as useful as on nuScenes. Moreover, the Waymo experiment has not been conducted, and it makes it therefore complicated to assess the overall usefulness of the pipeline.
The PDF also has not been updated, this makes it hard to truly assess the changes the paper underwent over the rebuttal and give an idea of what the final paper would look like.
Therefore, I will keep my score as weak reject.
I thank the authors for their rebuttal and replies.
Thank you for the detailed feedback. We agree that the presentation can be improved, particularly in explaining the motivation and making the method self-contained. We will substantially streamline the manuscript, simplify the terminology, and reorganize the method section to make the main ideas and implementation easier to follow.
Motivation and Fig. 2. We will clarify that the caption-alignment experiment is a diagnostic study used to motivate VISA, rather than a component of our proposed method. Its purpose is to examine whether improved feature–text alignment translates into better closed-set occupancy prediction. While the auxiliary text-alignment metrics improve, the downstream occupancy metrics do not, motivating our use of structured, instance-grounded semantic supervision. We will reorganize this discussion around that central observation and simplify Fig. 2 and its caption, including clearer definitions of the alignment and retrieval metrics.
Method details in the main paper. We will make the main method description more self-contained by consolidating the audit generation, crop/view selection, object-to-voxel grounding, and training objectives in the main text. We will also explicitly define the binary visual attribute vector, the predefined attribute vocabulary, and the class-to-attribute matrix, clarifying how instance-level VLM attributes are converted into supervision for occupancy predictions. The supplementary material will retain detailed implementation settings and additional analyses.
Reference correction and proofreading. Thank you for identifying the incorrect RenderOcc citation. The correct venue is ICRA 2024, and we will fix the reference. We will also carefully review the manuscript for terminology, notation, figure labels, references, and consistency between reported results and experimental records.
Thank you for pointing this out. We provide the VLM audit prompt and attribute vocabulary in Appendix B, under "Blind VLM audit prompt," but agree that their relationship could be clearer.
VISA uses a fixed vocabulary of 10 binary visual attributes for nuScenes, shared across all object instances. These attributes describe visually observable properties, such as
large_vehicle,long_body, andtwo_wheeler, selected to capture semantic distinctions relevant to the occupancy taxonomy. The VLM predicts an instance-specific binary attribute vector from each object crop, rather than deriving it directly from the ground-truth class.The predefined vocabulary is also used to construct the fixed class-to-attribute matrix for training. We will clarify how the vocabulary is specified in the VLM prompt, explicitly define the attribute vector and matrix in the main text, and present the complete prompt and output schema together to avoid ambiguity.
Thank you for the suggestion. We additionally evaluated VISA using Molmo2-7B and GLM-4.1V-9B, alongside the original Qwen3-VL-8B auditor. Due to the cost of auditing the full dataset with a proprietary API-based model such as ChatGPT, we included GLM-4.1V-9B as an additional alternative VLM.
The values in parentheses indicate absolute mIoU improvements over the occupancy-only baseline. All three VLM auditors improve overall semantic mIoU, with gains ranging from +0.50 to +0.70 points. These results support the applicability of VISA across different VLM teachers, rather than relying on a specific auditor.