acceptodds
Withdrawn from ICLR 2027

VGSSC: Surface-Aware Voxel–Gaussian Representations for Monocular Semantic Scene Completion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Monocular semantic scene completion seeks to infer a complete semantic occupancy field from a single RGB observation, requiring precise reconstruction of observed surfaces and reliable completion of unseen scene regions. This challenge becomes particularly pronounced in cluttered indoor scenes, where local geometry is complex and large portions of the scene remain unobserved. Existing methods typically represent scenes using either voxels or Gaussian primitives. Voxels preserve location-specific information throughout the volume but often produce spatially fragmented predictions, while Gaussians promote local geometric continuity yet may blur semantic boundaries when primitive supports overlap. We introduce VGSSC, a surface-aware Voxel–Gaussian framework that combines geometry-guided feature sharing with location-specific volumetric reasoning. Specifically, we introduce surface-relative distance tokens to encode the signed depth offset of each voxel from the predicted surface, providing explicit geometric cues that distinguish free space, near-surface regions, and occluded space. These surface-relative cues are incorporated into voxel features and also guide the initialization of surface-anchored Gaussian primitives. A Gaussian-assisted voxel decoder subsequently transfers locally aggregated geometric and semantic information from the Gaussian representation back to the dense volumetric, enabling volumetric completion beyond directly observed regions. Experiments on Occ-ScanNet and EmbodiedOcc-ScanNet demonstrate that VGSSC achieves state-of-the-art performance, including a 5.28% mIoU improvement over the previous method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Semantic Occupancy Prediction, Indoor Scene Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QuaaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VGSSC addresses monocular indoor semantic scene completion using a surface-aware Voxel–Gaussian representation. It encodes signed voxel-to-surface depth offsets as learnable tokens, initializes Gaussians from image and voxel features, and transfers refined Gaussian information back to a voxel decoder. Training-free probability fusion extends the model to online prediction. Experiments on Occ-ScanNet and EmbodiedOcc-ScanNet include component ablations and backbone and efficiency comparisons.

Critical strengths
  1. Using the predicted surface as a common reference gives the hybrid representation a clear rationale. Distance tokens distinguish voxels sharing image features, while Gaussians incorporate voxel context and return locally aggregated information to the voxel field.
  2. The reported improvements are substantial. Removing distance encoding or Gaussian assistance reduces single-frame mIoU by 1.76 and 2.00 points, supporting the usefulness of both components.
Critical weaknesses
  1. The central fusion mechanism remains underspecified. Equation (7) does not explain the fusion architecture, and Appendix A.2 does not fully specify how support-weight gating combines the feature and semantic streams. Gaussian anchor count and sampling, and the online observation-weight function, also remain unclear. These omissions affect reproduction of the core method and online results.
  2. The distance-encoding comparison is difficult to assess because the Continuous Offset implementation and Fixed Scale value are unspecified. The unexplained geometric IoU drop for Continuous Offset further weakens the evidence supporting the discrete-token design, although it does not establish an experimental error.
Questions for authors
  1. Could you provide the complete fusion architecture, including support-weight gating, and specify the Gaussian anchor count and sampling strategy?
  2. How are the Continuous Offset and Fixed Scale baselines implemented? Please confirm and explain the Continuous Offset result of 56.54 IoU / 56.20 mIoU, compared with 64.80 / 55.40 without distance encoding.
  3. What is the exact formula for the online observation weight ωn(v)\omega_n(v), including confidence estimation and free-space/behind-surface weighting?
  4. Since VGSSC without Gaussian assistance already achieves 55.16 mIoU, compared with 51.88 for GPOcc, could the authors clarify whether the improvements are evaluated under matched backbone, pretraining, and depth supervision settings?

If the authors can address most of my concerns, I would be willing to raise my score.

Reviewer_4TysOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VGSSC, a hybrid voxel-Gaussian framework for monocular semantic scene completion. Predicted depth provides a shared surface reference for signed distance-token encoding in the voxel branch and surface-anchored Gaussian initialization. Gaussian features are subsequently transferred to a dense voxel representation and refined for semantic occupancy prediction. Experiments on Occ-ScanNet and EmbodiedOcc-ScanNet report improved single-frame and online completion, with the latter obtained through probability fusion without additional temporal training.

Critical strengths
  1. The surface-relative distance encoding is a useful design for distinguishing voxels that share an image projection but occupy different positions along the ray. Using the same predicted surface to initialize Gaussians gives the two representations a coherent geometric reference and connects local Gaussian aggregation with dense volumetric reasoning.
  2. The reported gains are substantial: Occ-ScanNet mIoU increases from 51.88% for GPOcc to 57.16%, a gain of 5.28 percentage points. The component ablations support the benefits of distance encoding, Gaussian assistance, and surface anchoring. The online results further show that the single-frame model can be extended through prediction-level fusion.
Critical weaknesses
  1. Sensitivity to the shared depth reference is not sufficiently evaluated. Both voxel distance tokens and Gaussian anchors depend on predicted depth, so an erroneous surface can affect both branches simultaneously. Table 6 replaces DAv2 with VGGT, but also increases model size from 100.56M to 944.92M parameters. As the paper acknowledges, this does not isolate depth accuracy. A controlled depth perturbation or diagnostic ground-truth-depth experiment would help quantify this dependency; the current results do not establish whether errors are amplified or mitigated by the hybrid representation.
  2. Several details needed to reproduce the hybrid and online mechanisms remain unclear. The paper describes back-projecting selected image locations without specifying the Gaussian count or selection strategy. Although Appendix A.2 defines Gaussian support weights and aggregation, it does not fully specify the voxel-Gaussian fusion operator. The online observation weight is also described qualitatively rather than by a complete formula. These choices can materially affect both accuracy and runtime.

The efficiency trade-off deserves clearer discussion. Appendix Table 5 reports 17.21 FPS versus 29.35 FPS for GPOcc on the same RTX 4090. A component breakdown would clarify the runtime cost accompanying the accuracy gains. Table 4 also reports 56.54% IoU and 56.20% mIoU for continuous offsets. Compared with 64.80% IoU without distance encoding, this is a decrease of 8.26 percentage points. The authors should analyze why continuous offset encoding causes such a substantial degradation in geometric completion.

Questions for authors
  1. With the image feature extractor fixed, how sensitive is completion to controlled depth errors? Could ground-truth depth or a near-surface/occluded-region breakdown clarify the dependence of each branch on the shared surface reference?
  2. Please specify the Gaussian count and pixel-selection strategy, the exact voxel-Gaussian fusion operation, and the complete online observation-weight formula with its hyperparameters.
  3. Which components explain the throughput gap to GPOcc under the reported hardware setting? Please provide a latency breakdown and analyze why continuous offset encoding in Table 4 yields an IoU 8.26 percentage points lower than using no distance encoding, to clarify the effect of the encoding choice.
Reviewer_KXQCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Summary This paper proposes VGSSC, a surface-aware Voxel–Gaussian framework for monocular semantic scene completion. The method combines surface-relative distance encoding, surface-anchored Gaussian modeling, and Gaussian-assisted voxel refinement to leverage the complementary strengths of dense voxel representations and adaptive Gaussian primitives. Experimental results on Occ-ScanNet and EmbodiedOcc-ScanNet demonstrate strong performance, including a 5.28 percentage-point improvement in mIoU over the strongest reported baseline on Occ-ScanNet. Overall, the paper presents a reasonable and effective framework. However, several concerns regarding conceptual novelty, the justification of the proposed representation, and the attribution of performance improvements limit its contribution.

Critical strengths

The proposed method achieves substantial improvements over existing state-of-the-art approaches on indoor semantic scene completion benchmarks. In particular, VGSSC improves both geometric occupancy IoU and semantic mIoU, with consistent improvements across semantic categories in the single-frame setting. These results demonstrate the practical effectiveness of the proposed framework.

Critical weaknesses

1.This work introduces signed surface-relative depth tokens into voxel lifting and combines them with surface-anchored Gaussian-to-voxel reasoning to form a joint framework. It is not the first to propose surface awareness, nor the first to propose depth-aware voxel lifting, nor the first to combine Gaussian with voxel/triplane representations.、 2.Do Gaussians improve deeply occluded regions or only near-surface geometry? 3.The ablation study lacks a base model that removes both Distance Token and Gaussian Assist simultaneously, as well as a model that retains only ordinary Gaussian fusion without sharing the surface-aware voxel context. 4.The final performance of VGSSC is highly dependent on the upstream visual geometry representation, so it is necessary to more strictly distinguish the contribution of the geometry prior from the contribution of the representation method. 5.In Table 4, Continuous Offset compared with No Distance Encoding: mIoU increases by 0.80 percentage points. Overall IoU decreases by 8.26 percentage points. A reasonable explanation needs to be provided.

Questions for authors

1.This work introduces signed surface-relative depth tokens into voxel lifting and combines them with surface-anchored Gaussian-to-voxel reasoning to form a joint framework. It is not the first to propose surface awareness, nor the first to propose depth-aware voxel lifting, nor the first to combine Gaussian with voxel representations. 2.Do Gaussians improve deeply occluded regions or only near-surface geometry? 3.The ablation study lacks a base model that removes both Distance Token and Gaussian Assist simultaneously, as well as a model that retains only ordinary Gaussian fusion without sharing the surface-aware voxel context. 4.The final performance of VGSSC is highly dependent on the upstream visual geometry representation, so it is necessary to more strictly distinguish the contribution of the geometry prior from the contribution of the representation method. 5.In Table 4, Continuous Offset compared with No Distance Encoding: mIoU increases by 0.80 percentage points. Overall IoU decreases by 8.26 percentage points. A reasonable explanation needs to be provided.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.