- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents FM-Occ, an image-based 3D semantic occupancy prediction framework targeting two failure modes: feature homogenization during sparse 3D upsampling and semantic degradation across decoder layers. To address these issues, the paper proposes (1) 3D Sparse Subvoxel Shuffle (3S), which generates child features in the channel dimension and rearranges them onto a finer sparse grid, and (2) Entropy-Regulated Layer-wise Memory (ER-LM), which maintains recurrent cross-layer memory and modulates it using prediction entropy. A class-weighted entropy loss, , is additionally introduced to improve semantic supervision. Experiments on SemanticKITTI show improvements in spatial feature variation and semantic occupancy performance, while additional experiments on DISC demonstrate that ER-LM can also improve a different BEV-based occupancy framework.
Critical strengths
-
The paper identifies and analyzes two concrete failure modes in sparse semantic occupancy prediction. In particular, the analysis of feature homogenization during sparse upsampling and semantic degradation across decoder layers provides useful empirical insight beyond simply reporting final occupancy accuracy. The increase in spatial CV from to with 3S and the reduction in mean semantic degradation from to with ER-LM + provide quantitative evidence that the proposed modules affect the targeted behaviors. I found this diagnostic perspective informative, as it attempts to connect architectural modifications to measurable intermediate failure modes rather than treating them purely as performance-enhancing components.
-
ER-LM shows reasonably consistent empirical effectiveness and some evidence of generalizability. ER-LM improves the baseline mIoU from to , and ER-LM combined with further improves it to . More importantly, ER-LM is also evaluated on the substantially different BEV-based DISC framework, where it improves mIoU from to . This cross-framework result makes the semantic-memory component more convincing than an architecture-specific modification.
Critical weaknesses
-
The methodological novelty of 3S is insufficiently distinguished from existing 3D point-cloud/sparse upsampling techniques. The paper itself acknowledges channel-to-space rearrangement as an established upsampling mechanism and characterizes 3S as combining channel-to-space feature rearrangement with matched coordinate expansion. However, closely related operations have already been explored in the 3D point-cloud literature. For example, sub-point convolution in D-PCC (He et al., "Density-Preserving Deep Point Cloud Compression", CVPR2022) expands an representation into channels corresponding to multiple child features and uses a periodic-shuffle-style operation to obtain an upsampled representation. In addition, parent-to-child coordinate expansion, including the generation and occupancy prediction of the children of a voxel, is a standard operation in multiscale/octree-based point-cloud geometry coding. These operations appear conceptually close to the core transformation in 3S,
N \times C_{\mathrm{in}}\ \rightarrow\ N \times (r^3 C_{\mathrm{out}})\ \rightarrow\ Nr^3 \times C_{\mathrm{out}},\$$ followed by matched child-coordinate expansion. The paper does show that its particular organization, especially normalization/activation before shuffling, performs better than several alternatives, but it remains unclear what the fundamental methodological novelty of 3S is beyond this particular organization of established operations. Since 3S is presented as one of the paper's main contributions, I consider the lack of a careful comparison with closely related sparse 3D / point-cloud upsampling operators a significant weakness. -
The ablations do not isolate whether the gain of ER-LM actually comes from entropy-based regulation rather than recurrent cross-layer memory itself. The paper compares ER-LM with EMA-based historical aggregation, but a key baseline appears to be missing: the same recurrent/LSTM memory mechanism without entropy regulation. Consequently, the current experiments establish that learnable recurrent memory is more effective than simple EMA, but do not establish that the proposed entropy-regulation mechanism is necessary for the observed improvement. A particularly important comparison would be
\text{No memory}\ ;\rightarrow;\ \text{EMA}\ ;\rightarrow;\ \text{plain recurrent/LSTM memory}\ ;\rightarrow;\ \text{entropy-regulated recurrent memory (ER-LM)}.\$$ Without the third configuration, the contributions of recurrent memory and entropy regulation are confounded. This is consequential because entropy regulation, rather than merely adding cross-layer recurrent memory, is the central distinguishing mechanism claimed by ER-LM.
Questions for authors
-
Clarification regarding the novelty of 3S. Could the authors clarify the technical distinction between 3S and prior 3D point-cloud/sparse upsampling operators based on child-feature generation and feature rearrangement, such as sub-point convolution and related multiscale point-cloud geometry reconstruction methods? In particular, is the main novelty of 3S the transformation
N \times C_{\mathrm{in}}\ \rightarrow\ N \times (r^3 C_{\mathrm{out}})\ \rightarrow\ Nr^3 \times C_{\mathrm{out}},\$$ the matched expansion of parent coordinates into $r^3$ child coordinates, or specifically the ordering of joint normalization/activation before the channel-to-space rearrangement? A precise operation-level comparison with the closest 3D sparse/point-cloud upsampling methods would materially affect my assessment of the novelty of 3S. -
Clarification regarding the contribution of entropy regulation in ER-LM. Have the authors evaluated a version of ER-LM that retains the same recurrent/LSTM cross-layer memory but removes entropy-based regulation (e.g., using a fixed or entropy-independent memory modulation)? If so, how does it compare with the full ER-LM? This comparison would help determine whether the reported improvement primarily results from introducing recurrent cross-layer memory or specifically from the proposed entropy-regulation mechanism.
-
Clarification regarding the sparse transposed-convolution baseline. Does the sparse transposed-convolution baseline generate exactly the same set and number of active output voxels as 3S at the stage where spatial CV is measured? In particular, is the baseline a generative sparse transposed convolution, or does it operate on a predefined target coordinate set? If the active output coordinate sets differ, part of the reported difference in spatial CV could potentially reflect differences in the sampled active voxel set rather than only differences in feature construction. Reporting the input/output active voxel counts and clarifying the coordinate-generation procedure for both operators would help interpret the versus CV comparison.