- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses how many past frames to feed per sample in camera-LiDAR 3D semantic occupancy prediction, with the decision made outside the predictor. The proposed Composite Scene Entropy (CSE) is computed from raw inputs only, as a weighted sum of the point-distribution entropy of the accumulated point cloud, the local-covariance-based DECI, and the gradient-histogram entropy of the camera images. The window length is increased from 1 to 16, and the first length at which the relative CSE increment falls below 0.03 is selected and fed once to the predictor. With SAGFormer on nuScenes-SurroundOcc, the paper reports 34.79 mIoU with 9 frames on average (Fixed-9: 33.35, Fixed-16: 34.83) and a model-side latency of 379 ms vs. 803 ms, and it reuses the same calibration for SDGOcc, DAOcc, and SSCBench-KITTI-360.
Critical strengths
-
Clean decoupling and inference-time plug-and-play capability: Decoupling temporal window selection from the fusion architecture via raw-input entropy saturation is conceptually neat and practically appealing. Notably, the authors demonstrate that it works effectively at inference time without requiring predictor modifications or retraining, improving mIoU over Fixed-9.
-
The experiments effectively separate the benefit of assigning longer windows to complex scenes from the effect of average temporal budget. Controls such as "Shuffled CSE Scores"—which maintains the same window distribution but breaks scene correspondence, dropping mIoU from 34.79 to 33.71 (Table 4)—alongside frozen-checkpoint evaluations convincingly show that performance gains stem from genuine scene-level adaptation rather than an increased average history length.
Critical weaknesses
-
Baseline choice for the efficiency claim. Since Adaptive (34.79 mIoU) does not outperform Fixed-16 (34.83 mIoU), the contribution rests on computational savings. However, the claimed 52.8% latency reduction is measured against a sliding-window Fixed-16, which appears to re-process all past frames at every step (latency grows from 289 ms for Fixed-9 to 803 ms for Fixed-16), rather than against streaming or feature-cached baselines such as SOLOFusion or StreamPETR. This matters for my assessment because, against such baselines, the efficiency gain, which is the paper’s main contribution, may shrink substantially.
-
Selection overhead and worst-case latency. Determining the window length requires loading and aligning all 16 candidate frames and computing CSE upfront, which adds a 101 ms selector overhead. For the 13.74% of samples assigned 13–16 frames (Table 11), the latency would be roughly Fixed-16 plus the selector (up to about 904 ms by my estimate), i.e., worse than Fixed-16 itself. This matters because real-time occupancy prediction is usually constrained by worst-case latency, not by the average.
-
Scene-level window and moving objects. A single scene-level window does not account for the different temporal needs of static background and moving objects. With SAGFormer, Adaptive improves all object classes over Fixed-16 but lowers all background classes (e.g., driveable surface 50.32 → 49.85; Table 2). In addition, as the authors note in the Limitations, accumulating points in the ego frame can leave stale geometry from moving objects, which may inflate the entropy and lead to unnecessarily long windows in dynamic scenes. An analysis of window allocation in dynamic scenes would help. This matters because it questions whether a scene-level decision, the core design choice, is the right granularity.
Questions for authors
Please see the questions in the Weaknesses.
Regarding C1:
Thank you very much for raising this question. The streaming or feature-caching baselines you mentioned have given us great inspiration. After carefully reviewing this and designing experiments for validation, we found that, of the 101 ms overhead in computing the dynamic historical frame windows in our method, the computation of global and local entropy for point clouds accounts for about 20 ms, while the image entropy part accounts for 81 ms. In our CSE framework, the global and local entropy of point clouds must be recomputed for each scene due to ego-motion, whereas the image entropy part can be cached! On the basis of caching image entropy, we only need to compute the input image entropy of the six views at the current timestep, which can reduce the overall overhead to about 14 ms. Therefore, by caching the image entropy results of historical frames, we can reduce the overhead of the history selector to about 34 ms.
Indeed, we previously did not fully consider streaming or feature-caching baselines. This is mainly because, in occupancy prediction task scenarios—especially in high-performing multimodal occupancy prediction methods—a fairly effective empirical finding is that "directly feeding stacked historical frame point clouds to the perception model works best." Although historical frame point clouds can introduce artifacts for dynamic objects, in occupancy scenes dynamic objects occupy only a minority of the area, while most areas are static roads, buildings, and so on. The gain from completing static regions by directly stacking historical frames is far greater than the loss caused by dynamic-object artifacts.
Therefore, we still maintain that the scenario we consider is valuable, at least in the field of occupancy prediction, while we also acknowledge that the current methodological framework presumably cannot be directly applied to tasks such as 3D object detection.
Regarding C2:
It is indeed true that, as you mentioned, all 16 candidate frames must be aligned. However, as I pointed out in my previous response, once we choose to cache the gradient histograms of historical images, the overall overhead of the selector can be reduced to 34 ms.
Second, Fixed-16 SAGFormer costs 879 ms, not 904 ms. Overhead depends on the occupancy predictor: SAGFormer’s memory/time grow rapidly with history because stacked frames inflate scene queries (~12k for one frame vs. ~95k for 16), heavily loading the transformer. In Table 4 of the revised paper, we added DAOcc+CSE and SDGOcc+CSE (BEV-based, not sparse-query). Without image-gradient-histogram caching, their E2E latency is only 217/206 ms; after subtracting CSE selector overhead, single-frame inference is 114/103 ms, and with fixed 16 frames their E2E cost is 314/308 ms. We initially chose SAGFormer because its query-based design makes memory/time change noticeably with history, making it a good illustrative example; for pure real-time consideration, we would use cached gradient histograms + a BEV-based occupancy predictor.
Therefore, we believe that from an engineering perspective, combined with image gradient histogram caching and a more efficient occupancy predictor, the real-time performance of the predictor can be guaranteed even in the worst case.
Regarding C3:
Yes, the limitation you raised—that our CSE cannot simultaneously accommodate the differing temporal requirements of dynamic backgrounds and moving objects—does indeed exist. Table 6 in the version we uploaded also fully illustrates this point: when a scene contains fast-moving objects, the average window size selected by CSE is only about 3.37 frames.
However, we still maintain that scene-level decision-making is reasonable and necessary for the occupancy prediction task. This is because occupancy prediction is also known as semantic scene completion, and this task itself is inherently scene-level. Of course, a side effect of this is that, as you expected, our method may not be transferable to 3D object detection scenarios (we also tried applying it directly to VoxelNeXt in a 3D object detection setting, and the performance was about 3 points lower in both L1 AP and APH than those reported in the VADet paper).
In summary, regardless of whether you are willing to change your current view after reading this response, we are very grateful for the comments you raised during this review process. From my experience across multiple submissions, you are clearly an expert in the field of 3D object detection, and likely an expert from industry as well. Moreover, during this review, you did not simply use an LLM tool to generate a pile of meaningless comments; instead, you carefully read our work. Please allow me to express my deep respect here for your careful reviewing. Regardless of the final outcome this work may experience, we will gladly accept it.