acceptodds
Under review as a conference paper at ICLR 2027

VISA: VLM-Guided Instance Semantic Auditing for 3D Semantic Occupancy Prediction

Abstract

Semantic 3D occupancy provides a compact scene representation for autonomous driving and robotic decision making, yet closed-set object semantics remain challenging for rare and visually confusable classes. A common way to transfer vision-language model (VLM) knowledge is to align 3D features with language embeddings; however, we find that such alignment can improve text-space objectives without reliably improving voxel-level semantic prediction. We introduce VISA, a task-aligned VLM supervision framework that uses an offline VLM as a structured semantic auditor rather than an embedding target. For each physical object instance, VISA produces a reliability-aware audit containing a closed-set class hypothesis, plausible confusions, and visual attributes, propagates the audit along the object track, and grounds it only to matched 3D object voxels. The resulting structured supervision is distilled directly into semantic logits, while the underlying occupancy architecture and inference pipeline remain unchanged. On nuScenes, VISA yields 3.75–5.72% relative mIoU improvements across two backbones, with 6.16–9.92% relative object-mIoU gains and 8.46–11.70% relative rare-class gains. Beyond autonomous driving, VISA achieves 1.47–2.68% relative mIoU improvements on NYUv2 across four monocular SSC architectures, extending its benefits to indoor scene completion from single RGB images without video input. These results support the applicability of VISA across outdoor and indoor settings, covering both temporal occupancy models and single-image scene-completion architectures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D semantic occupancy prediction, semantic scene completion, vision-language models, instance-level semantic auditing, world models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2hQaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

I would like to first clarify that no generative AI was used for review. The submitted paper has not been exposed to LLM analysis from my side. The current review has also not been cross-checked via LLM. This is in hope to keep the reviewing process as confidential as possible, while reflecting the opinion of the reviewer as much as possible.

The paper introduces VISA, a supervision framework that uses an offline VLM to improve the predictions of a 3D occupancy model. Before training, Qwen3-VL-8b receives crops of representative instances from the scene. It then audits each crop and outputs representative data such as class, confidence score. Each of these audits are attached to the same instance over the temporal timesteps. During training, the audited crops are used to provide further supervision to the occupancy prediction model through taxonomy, attribute and graph auditing scores. At inference, the occupancy network is queried without all the VLM scaffolding, essentially not inducing any cost.

Critical strengths
  • VISA bridges the world of vision-language models and occupancy prediction, which are two trendy and impactful domains of computer vision.
  • The method developed in this paper is sound and the performance gain from VISA clearly matches intuition.
  • Moreover, these performance gains are non negligible.
  • The idea of using a soft distribution blending thanks to VLM auditing is quite interesting. As far as I know this is an original approach (other VLM approaches supervise in a different way). The taxonomy loss that stems from it effectively repurposes a classic cross-entropy loss.
  • The method does not limit itself to simply blending the VLM distributions to the dataset distribution, but also uses other interesting auditing cues such as confusion classes, object attributes (although it severely lacks description, this is interesting), enriching the distillation process.
  • Since the auditing is performed offline, VISA induces no cost during during training.
Critical weaknesses

All that follows are, in my opinion, critical weaknesses.

Paper structure and clarity

  1. Clarity and conciseness. The paper is overall hard to read, uses a lot of unnecessary jargon and could be heavily simplified (I develop some examples in the following items). I would invite the authors to heavily polish the manuscript and simplify many of the parts, probably by rewriting them manually and clearly. This would heavily improve the impact of the manuscript.
  2. Better explanation of the motivation experiment. From L. 159 until L. 177, it is not clear that this loss is used as a pretext experiment to motivate the introduction of VISA. It is essential to make it clear. Moreover, L. 167-171 feel like an enumeration of different experiments rather than a point being made or a discussed result. The conclusion of these experiments only comes at L. 175, and everything written before is hard to understand. The figure 2 that accompanies this paragraph is also complicated to read "paired text cosine measures projected 3D feature–caption similarity" is supposed to be a single unit of text, but is not clear. This would improve readability and reproduction.
  3. Too much reference to the supplementary for the main method. Appendix A should be moved into the main paper such that readers can understand the manuscript on its own. This might come with shrinking some parts, yet, the problem formulation should remain clear. The "Offline VLM audits and Object Voxels" (L. 717) and "Learning objective" (L. 739) paragraphs are especially important and should not be relegated to the appendices. Supplementary B, especially the crop extraction and view selection, as well as the training losses should also be brought back to the main manuscript. Such reference is understandable for minor experimental details or additional experiments, but should not be used for main components of the method. Because of that, L.156 is confusing "In this section" actually refers to section 3 but sounds like it is referring to the appendix. This results in cases where variables introduced in the supplementary materials are used in the main manuscript without context (e.g. qiq_i). This makes reading especially hard. Similarly, it is not exactly clear what the attribute vector is (L. 231). Clarifying what it represents is important since it is used to created the class-to-attribute matrix M\mathbf{M} (L. 231), which is involved in the attribute loss term via u^i,t\mathbf{\hat{u}}_{i,t}.
  4. Hallucinated reference. The RenderOcc paper (M. Pan et al., 2024) is reported as CPVR, but has been published in ICRA 2024. This and other synthetic-feeling sentences hint towards a paper that has been improperly proofread and AI-generated. This plays against the quality of the work and I would urge authors to please recheck all their experimental results, references and writings. This will improve the quality of the manuscript and strengthen the submission.

Method clarification

  1. Specification of the visual attributes ui\textbf{u}_i. Referenced multiple times in the method section and defined in supplementary A, L. 722 and 724, it is still unclear what the "predefined visual attributes" are. It would seem they are binary answerable string such as "two_wheeler" (L. 233) but it is never explicitly stated. In that case, how are they made or selected? Deducting from the shape of M\mathbf{M}, there should be a total of AA attributes for each object. I also suppose them to be shared? But at no moment they are explicitly defined. I would kindly ask authors to clarify this point. The "Blind VLM audit prompt" (L. 762) which is used to prompt the VLM for auditing itself seems ambiguous: "binary visual attributes useful for 3D semantic occupancy" (L. 770) is the only information given to the model, although it is supposed to output these predefined attributes. Could authors precisely clarify this?

Experimental suggestions

  1. VLM ablation. in a similar spirit, ablations with at least one other VLM would be appreciated. Using Molmo2-7b would be interesting since it is different from Qwen. If credits allow, using the ChatGPT API would be an interesting experiment too, allowing to assess the difference between open- and closed-source models. This would once again help framing the performance of the method better.
  2. Strengthening the benchmark results. While the results of the main benchmark are appealing, it would be interesting to evaluate the approach at least with other widely used outdoor datasets, i.e., Waymo and KITTI. The concern that I have is that VISA is hard to adapt to other datasets. Could authors report results on these additional datasets? This would heavily strengthen the impact and scope of the method. It is also expected from deep learning approaches that they generalize well.

From here on, I detail minor weaknesses.

Minor weaknesses

  1. Reference format. Citations should be in parentheses. Please refer to the official ICLR format.
  2. Unclear figure 1. In figure 1, it is unclear how the two rows interact. Adding an arrow that goes from the second row to the first row might clarify it. It is also unclear if the "audit" solid green box is also exactly the "offline VLM auditor". Carefully finetuning that figure would improve the understanding of the method.
  3. Referring supplementary materials. Please indicate that the reference is in the supplementary material when referring to content that is located there (e.g. Eq. 13, L. 188).
  4. List of all closed-set classes. It would be appreciated if the authors could list all the closed-set classes used when querying the VLM (L. 188).
  5. Clarification on cropped images. Why are images qualified as "sparse" (L. 194), although they are semantically dense since cropped around the object of interest?
  6. Better function introduction. ⊮ should be introduced before, or right after use (L. 215). I think it is supposed to be an indicator function 1\mathbf{1}?
  7. Some sentences feel extremely synthetic and are not useful for understanding. Please, consider removing the following sentences "not treated as calibrated probabilities" (L.210). There are also a lot of sentences in the form "it is not X, but rather Y", in which one of the clause is not necessary and simply confuse reader more than anything (L. 234, 251, 409, 727, I will not list them all). This feels very synthetic in my opinion.
  8. Loss coefficients. Please specify the loss coefficients for the VISA loss in the main manuscript instead of the supplementary materials, so readers can understand what drives learning.
  9. Table 3 location. The gap between table 3c and section 4.4 looks strange. Perhaps moving the three tables to the top of the page might give render better.
  10. Closed-set agreement statistics. The closed-set agreement between the VLM and the training split of nuScenes is reported at 87.7% in table 4. This is very interesting. I would be curious to know if the 12.3% disagreement is more of a VLM failure or a nuScene annotation issue. I think investigating statistics like average loss on the disagreeing samples vs. average loss on the agreeing samples might be interesting to better understand how VISA drives and improves the learning.
  11. Main figure mislabeled. In figure 1, the VLM auditor box annotates the object with "track" instead of "truck".
Questions for authors

Please, refer to the weaknesses section.

In the current form, the weaknesses of the paper outweigh the strengths of the paper. Nonetheless, the core idea developed in VISA is interesting and worth investigation. I would invite authors to clearly assess the points raised in the weaknesses sections.

Specifically, I would like to see results of VISA on other outdoor datasets (KITTI, Waymo), showing that the method could be applied in more diverse scenarios. The ablation experiments would be more than appreciated, but not a priority in comparison to the KITTI and Waymo results. I would also like the manuscript to be heavily clarified.

Most of my comments are not experimental requests, therefore I hope that the authors can address all my concerns during the rebuttal period.

If all these issues are solved, I will consider raising my score to borderline accept or accept.

3 comments
  1. Paper clarity, motivation, and method presentation

Thank you for the detailed feedback. We agree that the presentation can be improved, particularly in explaining the motivation and making the method self-contained. We will substantially streamline the manuscript, simplify the terminology, and reorganize the method section to make the main ideas and implementation easier to follow.

Motivation and Fig. 2. We will clarify that the caption-alignment experiment is a diagnostic study used to motivate VISA, rather than a component of our proposed method. Its purpose is to examine whether improved feature–text alignment translates into better closed-set occupancy prediction. While the auxiliary text-alignment metrics improve, the downstream occupancy metrics do not, motivating our use of structured, instance-grounded semantic supervision. We will reorganize this discussion around that central observation and simplify Fig. 2 and its caption, including clearer definitions of the alignment and retrieval metrics.

Method details in the main paper. We will make the main method description more self-contained by consolidating the audit generation, crop/view selection, object-to-voxel grounding, and training objectives in the main text. We will also explicitly define the binary visual attribute vector, the predefined attribute vocabulary, and the class-to-attribute matrix, clarifying how instance-level VLM attributes are converted into supervision for occupancy predictions. The supplementary material will retain detailed implementation settings and additional analyses.

Reference correction and proofreading. Thank you for identifying the incorrect RenderOcc citation. The correct venue is ICRA 2024, and we will fix the reference. We will also carefully review the manuscript for terminology, notation, figure labels, references, and consistency between reported results and experimental records.

  1. Specification of visual attributes.

Thank you for pointing this out. We provide the VLM audit prompt and attribute vocabulary in Appendix B, under "Blind VLM audit prompt," but agree that their relationship could be clearer.

VISA uses a fixed vocabulary of 10 binary visual attributes for nuScenes, shared across all object instances. These attributes describe visually observable properties, such as large_vehicle, long_body, and two_wheeler, selected to capture semantic distinctions relevant to the occupancy taxonomy. The VLM predicts an instance-specific binary attribute vector from each object crop, rather than deriving it directly from the ground-truth class.

The predefined vocabulary is also used to construct the fixed class-to-attribute matrix for training. We will clarify how the vocabulary is specified in the VLM prompt, explicitly define the attribute vector and matrix in the main text, and present the complete prompt and output schema together to avoid ambiguity.

  1. Generalization across different VLM auditors.

Thank you for the suggestion. We additionally evaluated VISA using Molmo2-7B and GLM-4.1V-9B, alongside the original Qwen3-VL-8B auditor. Due to the cost of auditing the full dataset with a proprietary API-based model such as ChatGPT, we included GLM-4.1V-9B as an additional alternative VLM.

VLM auditorOcc. IoU ↑mIoU ↑Object mIoU ↑Rare mIoU ↑
Occupancy-only33.6421.0818.0115.72
Qwen3-VL-8B33.6721.78 (+0.70)18.6116.03
Molmo2-7B34.5221.64 (+0.56)18.0115.72
GLM-4.1V-9B34.7821.58 (+0.50)18.3315.95

The values in parentheses indicate absolute mIoU improvements over the occupancy-only baseline. All three VLM auditors improve overall semantic mIoU, with gains ranging from +0.50 to +0.70 points. These results support the applicability of VISA across different VLM teachers, rather than relying on a specific auditor.

  1. Additional outdoor benchmark: SemanticKITTI.

Thank you for suggesting additional outdoor evaluations. We further evaluated VISA on SemanticKITTI, using two monocular semantic scene completion architectures, MonoScene and ET-Former.

MethodOcc. IoU ↑mIoU ↑
MonoScene34.1611.08
+ VISA34.30 (+0.14)11.25 (+0.17, +1.53%)
ET-Former51.4916.30
+ VISA51.55 (+0.06)16.42 (+0.12, +0.74%)

Values in parentheses indicate absolute improvements in percentage points and, for mIoU, relative improvements. Unlike the six-camera nuScenes setting, these experiments use only a single RGB image, providing a complementary evaluation under more limited visual observations. VISA improves semantic mIoU on both architectures, with relative gains of 1.53% and 0.74%, respectively. Together with our nuScenes and NYUv2 results, these experiments extend VISA to three datasets across indoor and outdoor environments, further supporting its generalizability across different visual input settings and occupancy architectures.

5.Response to Minor Comments (8–18)

Thank you for these detailed suggestions. We will address the presentation, notation, and formatting issues in the revised manuscript.

  • References and notation (8, 10, 13): We will correct the citation formatting, explicitly indicate references to supplementary equations, and define the indicator function at its first use.
  • Figures and layout (9, 16, 18): We will revise Fig. 1 to clarify the connection between offline VLM auditing and training-time supervision, distinguish the auditor from its output audit, and correct "track" to "truck". We will also improve the placement of Table 3.
  • Terminology and writing (12, 14): We will replace the ambiguous use of "sparse images" with "object-centric crops" and simplify repetitive or unnecessarily complicated sentences throughout the manuscript.
  • Closed-set classes (11): We will explicitly list the ten nuScenes object classes used for VLM auditing: barrier, bicycle, bus, car, construction vehicle, motorcycle, pedestrian, traffic cone, trailer, and truck.
  • Loss coefficients (15): We will include the VISA loss coefficients in the main text for clarity.

Analysis of VLM–annotation disagreements (17).

We appreciate this suggestion. To better understand the 12.3% disagreement between VLM predictions and nuScenes annotations, we further analyzed the characteristics of agreeing and disagreeing audits.

MetricAgreeDisagree
Mean VLM class confidence0.6970.507
Mean visual reliability0.6520.497
Mean object distance20.30 m27.78 m
Median crop area18,416 px²9,285 px²
Poor visibility1.60%18.75%

Disagreeing audits tend to have lower confidence and visual reliability, greater object distances, and substantially smaller crops. Notably, poor visibility is nearly 12 times more frequent** among disagreements. These findings suggest that limited visual evidence is an important factor associated with VLM–annotation disagreements, further motivating VISA's reliability-aware filtering and weighting.

While these statistics cannot definitively distinguish VLM recognition errors from annotation ambiguities, they provide additional insight into when disagreements arise and why treating all VLM predictions as equally reliable supervision would be undesirable. We will incorporate this analysis into the revised manuscript.

Reviewer_2hQaOpenReview ↗

Reply assessment and final score.

Thank you for your replies.

The authors clarified and dissipated some of my concerns. I especially found the VLM ablation analysis particularly interesting, however I still have doubts related to the generalization of the approach.

Although the auditing information are quite comprehensive and should be able to improve the performance of the model on KITTI, gains are extremely marginal, this seems to imply that the method is not necessarily as useful as on nuScenes. Moreover, the Waymo experiment has not been conducted, and it makes it therefore complicated to assess the overall usefulness of the pipeline.

The PDF also has not been updated, this makes it hard to truly assess the changes the paper underwent over the rebuttal and give an idea of what the final paper would look like.

Therefore, I will keep my score as weak reject.

I thank the authors for their rebuttal and replies.

Reviewer_2GsjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper wants to use VLM knowledge to help closed-set 3D semantic occupancy, especially on rare and easily confused classes. It runs Qwen3-VL-8B offline on a crop of every annotated object. Three small auxiliary losses then act on the box-averaged class probabilities:

Inference is unchanged. On nuScenes, OccWorld goes from 19.06 to 20.15 mIoU and GaussianWorld from 21.33 to 22.13. On NYUv2, four monocular SSC models get 1.5–2.7% relative mIoU gains.

Critical strengths

The question is interesting. The diagnostic in Fig. 2, showing that better text alignment doesn’t mean better voxels, is a useful message for people trying to bolt CLIP-style alignment onto closed-set tasks.

It’s a pure training-time add-on with zero inference cost.

Critical weaknesses

The method is basically instance-level soft-label distillation, and there is no comparsion against other training-only plug-ins, such as GaussRender and RenderOcc.

Geometric IoU goes up a lot, and I don’t see why. This needs an explanation. Is it from the object-region term changing how the empty class is treated? Better geometry IoU may lead to better mIoU, but the improvment of mIoU is less.

The backbone choice and the baseline numbers need context. See Questions For Authors.

Questions for authors

The two nuScenes backbones are both reproduced, and why the reproduced GaussianWorld is below the official result?

Does VISA still help on a stronger single-frame model, such as GaussianFormer-2 or QuadricFormer, which are already in Table 1? Showing gains on the strongest available model would be more convincing.

Why IoU improves more than mIoU?

2 comments

1 Comparison with existing training-time supervision methods.

Thank you for raising this point. We would like to clarify that VISA goes beyond instance-level soft-label distillation, which accounts for only its taxonomy supervision component. Our main contribution is a structured approach to transferring VLM knowledge into closed-set 3D occupancy prediction: VISA combines instance-specific class ambiguity, visual attributes, and reliability estimates, grounds these audits to matched 3D object voxels, and jointly optimizes taxonomy, attribute-factor, and scene-level audit graph supervision. This allows us to exploit richer VLM semantic information while preserving the original occupancy architecture and inference pipeline.

To further validate its effectiveness, we conducted additional comparisons with GaussRender and RenderOcc-style supervision using GaussianWorld as the shared backbone. GaussRender and RenderOcc both employ 2D rendering-based supervision, but differ in their original training settings. GaussRender is designed as a plug-in that complements standard 3D occupancy losses, whereas RenderOcc primarily investigates learning 3D occupancy from 2D semantic and depth annotations without dense 3D supervision. To compare their supervision objectives under a shared backbone, we retain GaussianWorld's original 3D losses and incorporate RenderOcc's ray-based rendering losses as auxiliary supervision. We therefore refer to this adaptation as RenderOcc-style, rather than claiming to reproduce the original RenderOcc training protocol.

Training supervisionOcc. IoU ↑mIoU ↑Object mIoU ↑Rare mIoU ↑
Baseline33.6421.0818.0115.72
+ GaussRender32.9221.3818.4116.02
+ RenderOcc-style33.9321.6618.5416.20
+ VISA33.6721.7818.6116.03
+ GaussRender + VISA34.1222.2619.3816.94
+ RenderOcc-style + VISA33.9122.3919.2717.21

Compared with the occupancy-only baseline, VISA improves mIoU by 0.70 points, compared with 0.30 for GaussRender and 0.58 for RenderOcc-style supervision. Adding VISA to GaussRender further improves mIoU from 21.38 to 22.26, while adding it to RenderOcc-style supervision improves mIoU from 21.66 to 22.39. The corresponding rare-class mIoU also increases from 16.02 to 16.94 and from 16.20 to 17.21, respectively. These results highlight the complementary value of VISA's structured, instance-grounded semantic supervision beyond rendering-based objectives.

  1. Why does geometric IoU improve more than semantic mIoU?

We thank the reviewer for this question. To better understand the geometric improvement, we analyzed occupied-space precision and recall on the same 150 validation scenes using a matched pair of GaussianWorld and VISA checkpoints.

MetricGaussianWorld+ VISAChange
Occupancy IoU31.7733.66+1.89
Occupancy Precision60.8656.42−4.44
Occupancy Recall39.9345.48+5.55

The results show that the IoU improvement is primarily associated with substantially higher occupied-space recall, recovering more previously missed occupied voxels, despite a decrease in precision.

This behavior is also consistent with VISA's supervision design. The taxonomy loss is applied to matched ground-truth object voxels, where the target distribution assigns zero probability to the empty class. Since the empty channel participates in the softmax, this provides an additional training signal favoring occupied predictions in these regions. While this does not isolate the contribution of the taxonomy loss, it offers a plausible explanation for the observed recall increase.

Importantly, occupancy IoU only requires correctly identifying occupied space, whereas semantic mIoU additionally requires assigning the correct class. Thus, improved recovery of occupied voxels can produce larger gains in geometric IoU than in class-wise semantic mIoU.

  1. GaussianWorld baseline reproduction.

The official GaussianWorld evaluation uses 30-frame sequences, whereas our reproduction uses 16-frame sequences due to GPU memory constraints. As also demonstrated in the original GaussianWorld paper (Fig. 5), sequence length affects streaming occupancy performance, with longer sequences generally improving mIoU. This difference in temporal context may therefore partly explain the lower reproduced baseline.

Importantly, we evaluate GaussianWorld and GaussianWorld + VISA under the same training and evaluation protocol. We also report results over three independent paired training seeds (see our response to Reviewer AdAy), with VISA improving over the baseline in all three runs, achieving an average gain of +0.80 mIoU points. This supports the effectiveness of VISA under a consistent evaluation setting.

  1. Generalization to stronger single-frame occupancy models.

Thank you for the suggestion. We additionally evaluated VISA on GaussianFormer-2 and QuadricFormer, two stronger single-frame occupancy models on nuScenes.

MethodOcc. IoU ↑mIoU ↑Object mIoU ↑Rare mIoU ↑
GaussianFormer-231.6520.7018.1015.90
+ VISA31.66 (+0.01)20.87 (+0.17)18.34 (+0.24)16.27 (+0.37)
QuadricFormer32.2121.0418.4816.20
+ VISA32.12 (−0.09)21.29 (+0.25)18.64 (+0.16)16.69 (+0.49)

VISA improves semantic mIoU on both models, with particularly encouraging gains in rare-class mIoU (+0.37 and +0.49). Together with our existing NYUv2 experiments across four single-frame monocular SSC architectures, these results further demonstrate that VISA generalizes across both single-frame and temporal occupancy models, without relying on temporal information for its semantic auditing framework.

Reviewer_AdAyOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper propose VISA, a framework for 3D semantic occupancy prediction. An offline VLM analyzes representative object crops and produces structured audits containing class hypotheses, plausible confusions, visual attributes, and reliability estimates. These audits are grounded to matched object voxels using training-time annotations and converted into taxonomy, attribute-factor, and scene-level graph losses. The original occupancy supervision is retained, and neither the model architecture nor the inference pipeline requires modification.

Critical strengths

Practical integration without additional inference cost VISA improves existing occupancy models through auxiliary training objectives. Audits are generated offline and reused, requiring no VLM, text encoder, object tracking, or additional prediction module at deployment.

Consistent semantic improvements across architectures and settings VISA improves semantic mIoU on both outdoor occupancy backbones and all four indoor scene-completion architectures.

Critical weaknesses

VISA improves over the full-structure crop teacher by 0.37 mIoU points, while shuffling audit semantics or attributes within each class reduces performance by only 0.11 and 0.17 points, respectively. Moreover, Table 1 states that results include standard deviations over three seeds, whereas the displayed table contains only single values.

Could the authors provide the missing standard deviations and paired seed-wise improvements, particularly for the crop-teacher and audit-shuffling comparisons?

Contribution centered on auxiliary-supervision construction. The experiments demonstrate the usefulness of this pipeline, the approach relies on several task-specific design choices, including the attribute vocabulary, class-to-attribute mapping, reliability thresholds, and graph construction rules. Consequently, its applicability to new datasets and taxonomies may require substantial manual adaptation, which is not sufficiently characterized.

Which components and hyperparameters can be reused unchanged when transferring VISA to a new taxonomy, and how much manual specification or tuning is required? The NYUv2 adaptation would provide a useful concrete example.

Questions for authors

See Weakness

2 comments

Thank you for the constructive feedback. We address both concerns below.

1. Standard deviations and paired seed-wise results

We agree that Table 1's caption refers to standard deviations that were not displayed. We provide the three-run statistics and paired improvements below. Parentheses indicate the mIoU improvement of VISA over the corresponding baseline under the same seed.

MethodIoU (mean ± std)Run 1Run 2Run 3mIoU (mean ± std)Paired Δ (mean ± std)
OccWorld31.64 ± 1.1018.7919.1019.2919.06 ± 0.25—
+ VISA33.51 ± 1.8520.08 (+1.29)20.24 (+1.14)20.13 (+0.84)20.15 ± 0.08+1.09 ± 0.23
GaussianWorld32.00 ± 1.0821.3321.4021.2621.33 ± 0.07—
+ VISA34.51 ± 0.9821.91 (+0.58)22.03 (+0.64)22.45 (+1.18)22.13 ± 0.28+0.80 ± 0.33

VISA consistently improves semantic mIoU across all three paired runs on both backbones, with mean improvements of 1.09 ± 0.23 and 0.80 ± 0.33 points, respectively.

For the crop-teacher and shuffling comparisons, parentheses indicate Full VISA minus the corresponding control under the same seed.

MethodRun 1Run 2Run 3mIoU (mean ± std)Paired Δ (mean ± std)
Full VISA21.9122.0322.4522.13 ± 0.28—
Full-structure crop teacher21.74 (+0.17)21.77 (+0.26)21.77 (+0.68)21.76 ± 0.02+0.37 ± 0.27
Within-class semantic shuffling21.97 (−0.06)22.03 (+0.00)22.06 (+0.39)22.02 ± 0.05+0.11 ± 0.24
Within-class attribute shuffling21.92 (−0.01)21.97 (+0.06)21.99 (+0.46)21.96 ± 0.04+0.17 ± 0.25

Full VISA outperforms the crop teacher in all three runs. The within-class shuffling controls preserve class-level semantic information while disrupting instance correspondence, and both yield lower average performance. Together, these results support the benefit of structured, instance-grounded supervision. All statistics are computed before rounding.

2. Manual adaptation and transfer to NYUv2

The core VISA framework, including offline auditing, taxonomy/attribute/graph supervision, reliability weighting, and the unchanged inference pipeline, is reusable across datasets. Adapting to NYUv2 primarily involves three dataset-level specifications:

  • Taxonomy and attributes: Seven audited indoor categories (chair, bed, sofa, table, tvs, furniture, objects), eleven binary visual attributes, and a manually defined 7 × 11 class-to-attribute mapping.
  • Instance grounding: Connected components are extracted from training occupancy labels, with visible voxels projected into the RGB image to generate object crops and establish component-level supervision.
  • Configuration: NYUv2 uses taxonomy mixing α = 0.20, confidence/reliability thresholds of 0.20, and graph positive-overlap/negative-margin thresholds of 0.35/0.20.

Importantly, the taxonomy and attribute mapping are specified once per dataset, while the VLM automatically generates audits for individual instances. No manual instance-level audit annotation is required. The same supervision framework is integrated into four NYUv2 architectures without modifying their inference pipelines.

We will clarify these adaptation details, including loss coefficients and any dataset- or backbone-specific configuration changes, in the revised manuscript.

Thanks for author's rebuttal, I decide to keep my score at weak accept

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.