acceptodds
Under review as a conference paper at ICLR 2027

FM-Occ: Feature-Diverse Upsampling and Memory-Aware Decoding for 3D Occupancy Prediction

Abstract

3D semantic occupancy prediction estimates voxel-wise occupancy and semantic labels throughout a scene. Our experiments reveal that conventional sparse transposed-convolution upsampling yields poorly differentiated voxel responses and that successive decoder layers discard previously correct semantic predictions. To address these issues, we present Feature-Diverse Upsampling and Memory-Aware Decoding for 3D Occupancy Prediction (FM-Occ), comprising two components. First, 3D Sparse Subvoxel Shuffle (3S) mitigates feature homogenization by constructing candidate child features in the channel dimension, jointly transforming and normalizing them, and then rearranging them onto a finer sparse grid with explicit feature-coordinate correspondence. Second, Entropy-Regulated Layer-wise Memory (ER-LM) retains semantic information across decoder layers by maintaining a layer-wise memory for each query through gated recurrent updates and using prediction entropy to regulate the historical information propagated to the next layer. During training, a class-weighted entropy loss provides label-guided regularization for matched semantic queries. On SemanticKITTI, 3S increases the spatial coefficient of variation of normalized voxel-wise root-mean-square (RMS) responses from 0.41% to 4.03%, while ER-LM reduces semantic degradation and improves error correction across decoder layers. The combined FM-Occ improves mIoU from 12.16% to 12.73% with negligible memory overhead. ER-LM further improves mIoU in the BEV-based DISC framework from 16.78% to 17.17%, supporting its applicability across 3D voxel-based and BEV-based architectures.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D occupancy prediction, autonomous driving, sparse voxel upsampling, query-based decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vBNkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FM-Occ, an image-based 3D semantic occupancy prediction framework targeting two failure modes: feature homogenization during sparse 3D upsampling and semantic degradation across decoder layers. To address these issues, the paper proposes (1) 3D Sparse Subvoxel Shuffle (3S), which generates child features in the channel dimension and rearranges them onto a finer sparse grid, and (2) Entropy-Regulated Layer-wise Memory (ER-LM), which maintains recurrent cross-layer memory and modulates it using prediction entropy. A class-weighted entropy loss, Lcwe\mathcal{L}_{\mathrm{cwe}}, is additionally introduced to improve semantic supervision. Experiments on SemanticKITTI show improvements in spatial feature variation and semantic occupancy performance, while additional experiments on DISC demonstrate that ER-LM can also improve a different BEV-based occupancy framework.

Critical strengths
  1. The paper identifies and analyzes two concrete failure modes in sparse semantic occupancy prediction. In particular, the analysis of feature homogenization during sparse upsampling and semantic degradation across decoder layers provides useful empirical insight beyond simply reporting final occupancy accuracy. The increase in spatial CV from 0.410.41% to 4.034.03% with 3S and the reduction in mean semantic degradation from 21.0021.00% to 16.5016.50% with ER-LM + Lcwe\mathcal{L}_{\mathrm{cwe}} provide quantitative evidence that the proposed modules affect the targeted behaviors. I found this diagnostic perspective informative, as it attempts to connect architectural modifications to measurable intermediate failure modes rather than treating them purely as performance-enhancing components.

  2. ER-LM shows reasonably consistent empirical effectiveness and some evidence of generalizability. ER-LM improves the baseline mIoU from 12.1612.16% to 12.6712.67%, and ER-LM combined with Lcwe\mathcal{L}_{\mathrm{cwe}} further improves it to 12.8812.88%. More importantly, ER-LM is also evaluated on the substantially different BEV-based DISC framework, where it improves mIoU from 16.7816.78% to 17.1717.17%. This cross-framework result makes the semantic-memory component more convincing than an architecture-specific modification.

Critical weaknesses
  1. The methodological novelty of 3S is insufficiently distinguished from existing 3D point-cloud/sparse upsampling techniques. The paper itself acknowledges channel-to-space rearrangement as an established upsampling mechanism and characterizes 3S as combining channel-to-space feature rearrangement with matched coordinate expansion. However, closely related operations have already been explored in the 3D point-cloud literature. For example, sub-point convolution in D-PCC (He et al., "Density-Preserving Deep Point Cloud Compression", CVPR2022) expands an N×dinN \times d_{\mathrm{in}} representation into channels corresponding to multiple child features and uses a periodic-shuffle-style operation to obtain an upsampled UN×doutUN \times d_{\mathrm{out}} representation. In addition, parent-to-child coordinate expansion, including the generation and occupancy prediction of the 23=82^3=8 children of a voxel, is a standard operation in multiscale/octree-based point-cloud geometry coding. These operations appear conceptually close to the core transformation in 3S,

    N \times C_{\mathrm{in}}\ \rightarrow\ N \times (r^3 C_{\mathrm{out}})\ \rightarrow\ Nr^3 \times C_{\mathrm{out}},\$$ followed by matched child-coordinate expansion. The paper does show that its particular organization, especially normalization/activation before shuffling, performs better than several alternatives, but it remains unclear what the fundamental methodological novelty of 3S is beyond this particular organization of established operations. Since 3S is presented as one of the paper's main contributions, I consider the lack of a careful comparison with closely related sparse 3D / point-cloud upsampling operators a significant weakness.
  2. The ablations do not isolate whether the gain of ER-LM actually comes from entropy-based regulation rather than recurrent cross-layer memory itself. The paper compares ER-LM with EMA-based historical aggregation, but a key baseline appears to be missing: the same recurrent/LSTM memory mechanism without entropy regulation. Consequently, the current experiments establish that learnable recurrent memory is more effective than simple EMA, but do not establish that the proposed entropy-regulation mechanism is necessary for the observed improvement. A particularly important comparison would be

    \text{No memory}\ ;\rightarrow;\ \text{EMA}\ ;\rightarrow;\ \text{plain recurrent/LSTM memory}\ ;\rightarrow;\ \text{entropy-regulated recurrent memory (ER-LM)}.\$$ Without the third configuration, the contributions of recurrent memory and entropy regulation are confounded. This is consequential because entropy regulation, rather than merely adding cross-layer recurrent memory, is the central distinguishing mechanism claimed by ER-LM.
Questions for authors
  1. Clarification regarding the novelty of 3S. Could the authors clarify the technical distinction between 3S and prior 3D point-cloud/sparse upsampling operators based on child-feature generation and feature rearrangement, such as sub-point convolution and related multiscale point-cloud geometry reconstruction methods? In particular, is the main novelty of 3S the transformation

    N \times C_{\mathrm{in}}\ \rightarrow\ N \times (r^3 C_{\mathrm{out}})\ \rightarrow\ Nr^3 \times C_{\mathrm{out}},\$$ the matched expansion of parent coordinates into $r^3$ child coordinates, or specifically the ordering of joint normalization/activation before the channel-to-space rearrangement? A precise operation-level comparison with the closest 3D sparse/point-cloud upsampling methods would materially affect my assessment of the novelty of 3S.
  2. Clarification regarding the contribution of entropy regulation in ER-LM. Have the authors evaluated a version of ER-LM that retains the same recurrent/LSTM cross-layer memory but removes entropy-based regulation (e.g., using a fixed or entropy-independent memory modulation)? If so, how does it compare with the full ER-LM? This comparison would help determine whether the reported improvement primarily results from introducing recurrent cross-layer memory or specifically from the proposed entropy-regulation mechanism.

  3. Clarification regarding the sparse transposed-convolution baseline. Does the sparse transposed-convolution baseline generate exactly the same set and number of active output voxels as 3S at the stage where spatial CV is measured? In particular, is the baseline a generative sparse transposed convolution, or does it operate on a predefined target coordinate set? If the active output coordinate sets differ, part of the reported difference in spatial CV could potentially reflect differences in the sampled active voxel set rather than only differences in feature construction. Reporting the input/output active voxel counts and clarifying the coordinate-generation procedure for both operators would help interpret the 0.410.41% versus 4.034.03% CV comparison.

Reviewer_PvBEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents FM-Occ for sparse 3D semantic occupancy prediction, targeting two issues: feature homogenization during sparse 3D upsampling and semantic degradation across decoder layers. It proposes 3S (3D Sparse Subvoxel Shuffle), which constructs and jointly normalizes child voxel features in the channel dimension before spatial rearrangement, so that upsampled sparse features retain more local response variation. It also proposes ER-LM (Entropy-Regulated Layer-wise Memory), which maintains a gated LSTM memory across decoder stages and uses classification entropy to modulate memory propagation, together with a class-weighted entropy loss.

Critical strengths
  1. This paper identifies two specific and architecture-relevant issues in sparse 3D occupancy prediction: feature homogenization during sparse upsampling and semantic degradation across decoder layers.

  2. The proposed ER-LM module shows some transferability, improving the BEV-based DISC framework from 16.78% to 17.17% mIoU and from 45.54% to 45.67% IoU when inserted into it.

Critical weaknesses
  1. The reported mIoU of FM-Occ (12.73%) is substantially lower than several compared methods, and the paper does not explain whether this gap stems from the sparse voxel representation, different evaluation protocols, or other factors.

  2. It would be helpful to evaluate FM-Occ on KITTI-360, as the current experiments are limited to SemanticKITTI and additional large-scale benchmarks would better support the generalization claims.

  3. The paper emphasizes sparse 3D representations but does not report inference speed or memory usage, and the overall performance does not show a clear advantage over dense or BEV-based baselines, so the practical benefit of the sparse design remains unclear.

  4. The comparison does not include several recent semantic scene completion methods such as MonoOcc [1], SGN [2], HASSC [3], and CGFormer [4]; including these methods, or explaining why they are excluded, would make the empirical comparison more convincing.

  5. The comparison does not include recent sparse occupancy methods such as SparseOcc (CVPR 2024) [5] and the fully sparse SparseOcc (ECCV 2024) [6]; including these methods is necessary.

[1] ICRA 2024, MonoOcc: Digging into Monocular Semantic Occupancy Prediction.

[2] IEEE TIP 2024, Camera-Based 3D Semantic Scene Completion With Sparse Guidance Network.

[3] CVPR 2024, Not All Voxels Are Equal: Hardness-Aware Semantic Scene Completion with Self-Distillation.

[4] NeurIPS 2024, Context and Geometry Aware Voxel Transformer for Semantic Scene Completion.

[5] CVPR 2024, SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction.

[6] ECCV 2024, Fully Sparse 3D Occupancy Prediction.

Questions for authors

Please see weakness

Reviewer_HkL7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FM-Occ, a framework for vision-based 3D semantic occupancy prediction, addressing two issues: feature homogenization during sparse 3D upsampling and semantic degradation across decoder layers. To address these issues, the authors introduce 3D Sparse Subvoxel Shuffle (3S), which constructs and normalizes child-voxel features before spatial rearrangement, and Entropy-Regulated Layer-wise Memory (ER-LM), which maintains recurrent cross-layer memory regulated by prediction entropy. A class-weighted entropy loss is also introduced. On SemanticKITTI, the complete FM-Occ improves mIoU from 12.16% to 12.73% over the baseline, while ER-LM improves the BEV-based DISC framework from 16.78% to 17.17% mIoU.

Critical strengths

1.The paper identifies two meaningful problems in sparse occupancy prediction. The analysis of feature homogenization and cross-layer semantic degradation provides useful insights into intermediate network behavior, rather than focusing solely on final prediction accuracy.

2.The experimental analysis is relatively comprehensive. The authors investigate different normalization strategies, feature allocation variants, and historical information aggregation methods. The five-seed robustness evaluation further supports the effectiveness of individual modules.

3.ER-LM demonstrates some architectural transferability. Its improvement on the BEV-based DISC framework suggests that the memory mechanism may benefit architectures beyond the proposed sparse voxel-based framework.

Critical weaknesses

1.The contribution of entropy regulation in ER-LM is insufficiently isolated. The current experiments compare ER-LM with EMA-based aggregation but lack an LSTM baseline without entropy regulation. Therefore, it remains unclear whether the improvement comes primarily from entropy regulation or recurrent memory itself.

2.The proposed components do not exhibit consistently complementary effects. In Table 2, ER-LM combined with the class-weighted entropy loss achieves 12.88% mIoU, outperforming the complete FM-Occ (12.73%). Although the complete model achieves higher occupancy IoU, the reduction in semantic mIoU suggests a potential trade-off between the proposed components.

3.The practical advantages of the framework remain unclear. FM-Occ achieves substantially lower mIoU than several methods in Table 1. Although GPU memory consumption is reported, inference latency and computational complexity are missing, and evaluation is limited to SemanticKITTI.

Questions for authors

1.Have the authors evaluated an LSTM-based memory mechanism without entropy regulation? A comparison among EMA, plain LSTM, and entropy-regulated LSTM would help isolate the contribution of entropy regulation.

2.Why does the complete FM-Occ achieve lower mIoU than ER-LM combined with the entropy loss alone, despite improving occupancy IoU?

3.Could the authors report inference speed and computational complexity, and evaluate the method on an additional dataset? These results would help clarify the practical benefits of the sparse design and its generalization ability.

Reviewer_NHfyOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents FM-Occ, a framework for camera-based 3D semantic occupancy prediction that addresses two limitations in sparse voxel reconstruction and iterative Transformer decoding: insufficient spatial differentiation of upsampled features and degradation of previously correct semantic predictions across decoder layers. The proposed framework contains two main components: 3D Sparse Subvoxel Shuffle replaces conventional sparse transposed-convolution upsampling with channel expansion, joint normalization, and deterministic channel-to-space rearrangement. The objective is to generate more differentiated child-voxel representations while preserving explicit feature-coordinate correspondence. Entropy-Regulated Layer-wise Memory introduces an LSTM-based memory mechanism across decoder layers, with prediction entropy controlling the modulation of propagated memory states. A class-weighted entropy regularization loss is additionally introduced to encourage concentrated class distributions for matched semantic queries. Experiments on SemanticKITTI show that the full framework improves mIoU from 12.16% to 12.73%, with GPU memory usage increasing from 13.05 GB to 13.09 GB. The authors also report increased spatial variation in voxel feature magnitudes and reduced semantic degradation across decoder layers. Applying ER-LM to the BEV-based DISC framework improves mIoU from 16.78% to 17.17%.

Critical strengths
  1. Meaningful problem motivation supported by informative intermediate-representation analyses.
  2. Lightweight architectural modifications and encouraging cross-architecture applicability.
Critical weaknesses
  1. The proposed 3S relies on established operations, including channel expansion, normalization, and channel-to-space rearrangement. Its feature expansion and spatial rearrangement strategy also shares conceptual similarities with FDR-Occ [1]. The paper does not sufficiently clarify the novelty of 3S beyond existing spatial feature expansion techniques.
  2. Several relevant studies [1-4] from 2025–2026 are not adequately discussed. This makes it difficult to assess the novelty of the proposed methods relative to recent advances in sparse feature refinement and memory-based occupancy prediction.
  3. The contribution of entropy regulation is not isolated from the underlying LSTM memory. Moreover, the full model achieves only a 0.57-point mIoU improvement and underperforms the ER-LM + entropy loss configuration (12.73% vs. 12.88%), raising concerns about the complementarity and robustness of the proposed components.

[1] FDR-Occ: Factorized Dense Routing for Full-Spectrum 3D Occupancy Prediction, ECCV 2026

[2] Stcocc: Sparse spatial-temporal cascade renovation for 3d occupancy and scene flow prediction, CVPR 2025

[3] Occupancy learning with spatiotemporal memory, ICCV 2025

[4] Sparsity-Aware Voxel Attention and Foreground Modulation for 3D Semantic Scene Completion, CVPR 2026

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.