acceptodds
Under review as a conference paper at ICLR 2027

Saturation-Guided Temporal Context Allocation Before 3D Occupancy Prediction

Abstract

Scene-adaptive temporal context allocation enables 3D occupancy predictors to tailor the history length to each scene. Existing adaptive methods infer this demand from detections, predicted observability, learned features, or other predictor-generated signals. Coupling the selection rule to a particular perception architecture limits its direct transfer to other occupancy predictors, even on the same dataset. We ask whether the required history can instead be determined directly from raw camera–LiDAR observations. Our key observation is that the additional geometric coverage and visual diversity contributed by older frames saturate at scene-dependent rates. Based on this observation, we propose Composite Scene Entropy (CSE), a predictor-external proxy that quantifies this saturation over nested temporal windows. CSE combines global point-cloud entropy, local point-cloud entropy, and image-gradient entropy to characterize global geometric coverage, local structure, and visual diversity, respectively. We select the shortest window whose marginal CSE gain has saturated and feed it to an otherwise unchanged occupancy predictor. Extensive experiments on nuScenes-SurroundOcc show that one CSE calibration and its per-sample allocations can be reused across three compatible occupancy predictors; with SAGFormer, CSE improves Fixed-9 by 1.44 mIoU at the same average window length and nearly matches Fixed-16 with 52.8% lower model-side latency. On SSCBench-KITTI-360, an independent calibration yields the same CSE parameter values as on nuScenes, and CSE allocation remains within 0.03 mIoU of Fixed-16.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D semantic occupancy prediction, Temporal context allocation, Multimodal perception, Adaptive computation, Camera-LiDAR fusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MawxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses how many past frames to feed per sample in camera-LiDAR 3D semantic occupancy prediction, with the decision made outside the predictor. The proposed Composite Scene Entropy (CSE) is computed from raw inputs only, as a weighted sum of the point-distribution entropy of the accumulated point cloud, the local-covariance-based DECI, and the gradient-histogram entropy of the camera images. The window length is increased from 1 to 16, and the first length at which the relative CSE increment falls below 0.03 is selected and fed once to the predictor. With SAGFormer on nuScenes-SurroundOcc, the paper reports 34.79 mIoU with 9 frames on average (Fixed-9: 33.35, Fixed-16: 34.83) and a model-side latency of 379 ms vs. 803 ms, and it reuses the same calibration for SDGOcc, DAOcc, and SSCBench-KITTI-360.

Critical strengths
  • Clean decoupling and inference-time plug-and-play capability: Decoupling temporal window selection from the fusion architecture via raw-input entropy saturation is conceptually neat and practically appealing. Notably, the authors demonstrate that it works effectively at inference time without requiring predictor modifications or retraining, improving mIoU over Fixed-9.

  • The experiments effectively separate the benefit of assigning longer windows to complex scenes from the effect of average temporal budget. Controls such as "Shuffled CSE Scores"—which maintains the same window distribution but breaks scene correspondence, dropping mIoU from 34.79 to 33.71 (Table 4)—alongside frozen-checkpoint evaluations convincingly show that performance gains stem from genuine scene-level adaptation rather than an increased average history length.

Critical weaknesses
  • Baseline choice for the efficiency claim. Since Adaptive (34.79 mIoU) does not outperform Fixed-16 (34.83 mIoU), the contribution rests on computational savings. However, the claimed 52.8% latency reduction is measured against a sliding-window Fixed-16, which appears to re-process all past frames at every step (latency grows from 289 ms for Fixed-9 to 803 ms for Fixed-16), rather than against streaming or feature-cached baselines such as SOLOFusion or StreamPETR. This matters for my assessment because, against such baselines, the efficiency gain, which is the paper’s main contribution, may shrink substantially.

  • Selection overhead and worst-case latency. Determining the window length requires loading and aligning all 16 candidate frames and computing CSE upfront, which adds a 101 ms selector overhead. For the 13.74% of samples assigned 13–16 frames (Table 11), the latency would be roughly Fixed-16 plus the selector (up to about 904 ms by my estimate), i.e., worse than Fixed-16 itself. This matters because real-time occupancy prediction is usually constrained by worst-case latency, not by the average.

  • Scene-level window and moving objects. A single scene-level window does not account for the different temporal needs of static background and moving objects. With SAGFormer, Adaptive improves all object classes over Fixed-16 but lowers all background classes (e.g., driveable surface 50.32 → 49.85; Table 2). In addition, as the authors note in the Limitations, accumulating points in the ego frame can leave stale geometry from moving objects, which may inflate the entropy and lead to unnecessarily long windows in dynamic scenes. An analysis of window allocation in dynamic scenes would help. This matters because it questions whether a scene-level decision, the core design choice, is the right granularity.

Questions for authors

Please see the questions in the Weaknesses.

1 comment

Regarding C1:

Thank you very much for raising this question. The streaming or feature-caching baselines you mentioned have given us great inspiration. After carefully reviewing this and designing experiments for validation, we found that, of the 101 ms overhead in computing the dynamic historical frame windows in our method, the computation of global and local entropy for point clouds accounts for about 20 ms, while the image entropy part accounts for 81 ms. In our CSE framework, the global and local entropy of point clouds must be recomputed for each scene due to ego-motion, whereas the image entropy part can be cached! On the basis of caching image entropy, we only need to compute the input image entropy of the six views at the current timestep, which can reduce the overall overhead to about 14 ms. Therefore, by caching the image entropy results of historical frames, we can reduce the overhead of the history selector to about 34 ms.

Indeed, we previously did not fully consider streaming or feature-caching baselines. This is mainly because, in occupancy prediction task scenarios—especially in high-performing multimodal occupancy prediction methods—a fairly effective empirical finding is that "directly feeding stacked historical frame point clouds to the perception model works best." Although historical frame point clouds can introduce artifacts for dynamic objects, in occupancy scenes dynamic objects occupy only a minority of the area, while most areas are static roads, buildings, and so on. The gain from completing static regions by directly stacking historical frames is far greater than the loss caused by dynamic-object artifacts.

Therefore, we still maintain that the scenario we consider is valuable, at least in the field of occupancy prediction, while we also acknowledge that the current methodological framework presumably cannot be directly applied to tasks such as 3D object detection.

Regarding C2:

It is indeed true that, as you mentioned, all 16 candidate frames must be aligned. However, as I pointed out in my previous response, once we choose to cache the gradient histograms of historical images, the overall overhead of the selector can be reduced to 34 ms.

Second, Fixed-16 SAGFormer costs 879 ms, not 904 ms. Overhead depends on the occupancy predictor: SAGFormer’s memory/time grow rapidly with history because stacked frames inflate scene queries (~12k for one frame vs. ~95k for 16), heavily loading the transformer. In Table 4 of the revised paper, we added DAOcc+CSE and SDGOcc+CSE (BEV-based, not sparse-query). Without image-gradient-histogram caching, their E2E latency is only 217/206 ms; after subtracting CSE selector overhead, single-frame inference is 114/103 ms, and with fixed 16 frames their E2E cost is 314/308 ms. We initially chose SAGFormer because its query-based design makes memory/time change noticeably with history, making it a good illustrative example; for pure real-time consideration, we would use cached gradient histograms + a BEV-based occupancy predictor.

Therefore, we believe that from an engineering perspective, combined with image gradient histogram caching and a more efficient occupancy predictor, the real-time performance of the predictor can be guaranteed even in the worst case.

Regarding C3:

Yes, the limitation you raised—that our CSE cannot simultaneously accommodate the differing temporal requirements of dynamic backgrounds and moving objects—does indeed exist. Table 6 in the version we uploaded also fully illustrates this point: when a scene contains fast-moving objects, the average window size selected by CSE is only about 3.37 frames.

However, we still maintain that scene-level decision-making is reasonable and necessary for the occupancy prediction task. This is because occupancy prediction is also known as semantic scene completion, and this task itself is inherently scene-level. Of course, a side effect of this is that, as you expected, our method may not be transferable to 3D object detection scenarios (we also tried applying it directly to VoxelNeXt in a 3D object detection setting, and the performance was about 3 points lower in both L1 AP and APH than those reported in the VADet paper).

In summary, regardless of whether you are willing to change your current view after reading this response, we are very grateful for the comments you raised during this review process. From my experience across multiple submissions, you are clearly an expert in the field of 3D object detection, and likely an expert from industry as well. Moreover, during this review, you did not simply use an LLM tool to generate a pile of meaningless comments; instead, you carefully read our work. Please allow me to express my deep respect here for your careful reviewing. Regardless of the final outcome this work may experience, we will gladly accept it.

Reviewer_igSTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how much temporal history should be admitted to a 3D occupancy predictor, as opposed to how history is fused. Motivated by the observation that adaptive window-selection policies in prior work are coupled to a specific perception pipeline (detector outputs, predicted observability, learned features), the authors ask whether the required history length can be decided directly from raw camera–LiDAR observations, before any predictor computation. They propose Composite Scene Entropy (CSE), a weighted combination of (i) global voxel-occupancy entropy, (ii) a local point-cloud structure statistic (DECI), and (iii) image-gradient histogram entropy, computed over nested temporal windows. A saturation rule selects the shortest window whose relative marginal CSE gain falls below a threshold, and this window is fed to an otherwise unchanged predictor. Because the selector is predictor-external, one calibration and the per-sample assignments can be reused across compatible predictors. On nuScenes-SurroundOcc, with SAGFormer, CSE allocation improves Fixed-9 by 1.44 mIoU at a similar average budget and comes within 0.04 mIoU of Fixed-16 with 52.8% lower model-side latency; the same calibration transfers to SDGOcc and DAOcc; on SSCBench-KITTI-360, a separate calibration reportedly selects identical parameter values.

Critical strengths

S1. Clean problem decomposition and clear positioning. The distinction between temporal fusion and temporal allocation (Fig. 1, Table 1) is well drawn, and formulating allocation as a sample-level stopping problem at the raw-input interface (Sec. 3) is a genuinely useful reframing. The formalization of quality loss and cost (Eqs. 4–5) with a common timing boundary that includes selector overhead is more careful than what is typical in this literature.

S2. Training-free, predictor-external selector with practical value. The selector requires no detections, features, or retraining per predictor. The frozen-checkpoint experiment (Table 12) is particularly compelling from a deployment standpoint: applying CSE at inference time to a Fixed-9-trained SAGFormer already yields +1.40 mIoU, nearly matching the retrained adaptive variant (34.75 vs. 34.79). The end-to-end latency saving of ~50% at near-Fixed-16 accuracy (Table 14) is a concrete, well-measured contribution.

Critical weaknesses

W1. No empirical comparison with any existing adaptive-temporal method. The closest related works—NGOcc (voxel-level observability-based fusion), VADet—are discussed conceptually (Table 1) but never compared empirically in cost or accuracy. The paper's core selling point over predictor-internal adaptation is transferability and cost, yet no experiment quantifies this advantage against an adaptive baseline, even on a single predictor. At minimum, a comparison against NGOcc (or an explanation of its infeasibility) is needed for the claimed advantage to be substantiated rather than asserted.

W2. Calibration is performed on the evaluation split. The CSE weights and τs are calibrated using SAGFormer's validation mIoU on nuScenes (Sec. 5.1, Appendix C), and the headline results are then reported on the same validation split. While common in this benchmark (test labels are withheld), the headline +1.44 mIoU and the τs = 0.97 operating point are fit to the evaluation data. The nuScenes-mini recalibration transfer (Table 8) and the flat local landscape (Table 6, all within 0.38 mIoU) mitigate but do not eliminate this concern; a calibration on a disjoint subset (e.g., train-split scenes) evaluated on val would close it.

Questions for authors

Q1. Why is n = 1 never eligible? If G(2) = 0 (the second frame adds no coverage or diversity), the rule still returns n* = 2 (Sec. 4.2). Is this intentional, and how often does it occur in practice?

Q2. In Eq. (7), d_i is inversely related to the local differential entropy, so top-K_l selection retains the least compact (most degenerate/planar) voxels in H_l. Is this the intended direction, and how does H_l behave as new frames densify already-observed structures (which would change neighborhood covariances even without new coverage)?

Q3. How does CSE behave in highly dynamic scenes? The image-gradient entropy in particular may keep changing without any new static coverage being added (moving vehicles), potentially over-allocating; conversely, does stale ego-aligned geometry from moving objects ever cause under-allocation where longer history would help? A stratified analysis by dynamic-object content would substantially strengthen the mechanism story.

1 comment

Regarding W1:

The VADet method relies on 3D bounding boxes provided by a 3D detection method, as well as the velocity attribute that the detector additionally predicts for each 3D bbox. Such an attribute is by default unavailable in our task (unless we use GT bboxes, but doing so would amount to cheating in their favor). Moreover, neither VADet nor NGOcc has open-sourced the corresponding code and weights, so it is very difficult for us to reproduce the performance reported in their papers (with the assistance of GPT6-astra, our reproduction of VADet following the paper achieves an mAPH 2 points lower than the original on the detection task). Therefore, the only comparison we can make here is with the Occ3D dataset results reported in the NGOcc paper. These experimental results have been added to Table 4 of the newly submitted paper, and the details of the table are as follows:

Table: Performance and inference cost on Occ3D-nuScenes validation. C/L denote camera/LiDAR. NGOcc† mIoU and memory are quoted from the paper; 16 is its nominal temporal configuration, and mIoU uses camera-mask training. For CSE, camera masks are used during training and evaluation, and frame counts are means including the current frame. CSE memory is peak inference process memory in GB (1 GB = 1000 MB), and E2E latency includes input preparation, CSE selection, and prediction. Dashes mark unavailable values.

MethodMod.Input framesIoU ↑mIoU ↑Memory (GB) ↓E2E latency (ms) ↓
NGOcc†C16--45.107.900--
SAGFormer+CSEC+L986.4552.857.422879
DAOcc+CSEC+L986.7956.126.522217
SDGOcc+CSEC+L986.9956.447.011206

Regarding W2:

Thank you very much for raising this question. I believe that in Table 10 of the supplementary material, we have already conducted the relevant experiment of directly transferring the parameters calibrated on NuScenes-mini to the SSCBench-KITTI-360 dataset. I think this set of experiments should address the issue you mentioned: "a calibration on a disjoint subset (e.g., train-split scenes) evaluated on val."

Table: Fine-grained CSE calibration on nuScenes-mini and inference-time parameter transfer with fixed SAGFormer checkpoints. Both checkpoints are trained on their respective training splits using CSE weights (0.4, 0.6). Evaluation replaces the inference-time weights with (0.42, 0.58) from nuScenes-mini. Target recalibration refers to additional target-side tuning of these transferred inference parameters.

Calibration splitα_g candidatesβ_g candidatesSelected (α_g, β_g, γ_I, τ_s)
nuScenes-mini train{0.36, 0.38, 0.40, 0.42, 0.44, 0.46}{0.56, 0.58, 0.60, 0.62, 0.64, 0.66}(0.42, 0.58, 1.00, 0.97)
Evaluation splitCheckpoint training split / policyInference CSE parameter sourceTarget recalib.Avg. framesIoU ↑mIoU ↑
nuScenes validationnuScenes train / validation-calibrated CSEnuScenes-mini trainNo954.9134.69
SSCBench-KITTI-360KITTI-360 train / CSE weights (0.4, 0.6)nuScenes-mini train (transferred)No959.6227.08

Regarding Q1:

Thank you for pointing out this edge case. The current rule uses the single-frame CSE as the baseline and starts checking the stopping condition from the two-frame window. Therefore, if G(2) = 0, the algorithm indeed returns n* = 2 and does not fall back to the single-frame case. As for the practical situation, we conducted a rigorous statistical analysis on the NuScenes validation set. According to the statistics, the case G(2) = 0 actually accounts for only 3% of the scenes, and in these 3% of scenes the data collection vehicle was stationary and there were no movable objects in the surrounding scene. Therefore, in these 3% of scenes, aggregating the entirely static historical point clouds and image frames does not bring any benefit.

Regarding Q2:

Thank you for pointing out the direction of the local statistic. We use top-K_l to retain the flattest voxels in H_l because in our practical experiments we found that the greatest benefit brought by historical frame point clouds is filling in the planar holes in the original space, thereby reducing the perception difficulty of the occupancy predictor for these regions. Therefore, the metric we designed is intended to reflect the extent to which the added historical frame point clouds can reflect planar coverage.

Regarding Q3:

Thank you very much for pointing out this issue. To address this, we analyzed cases with sufficient history and fast-moving objects. For objects >5 m/s, our selector averages 3.37 frames; for >10 m/s, 3.36 frames. Details are in Table 6 of the revised main text, omitted here due to length limits.

Reviewer_jDExOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to decouple the temporal context allocation of 3d semantic occupancy prediction from the perception pipeline. It proposes Composite Scene Entropy (CSE), which combines global geometric coverage, local geometric compactness statistic, and image-gradient entropy. The reported results improve mIoU from 33.35 to 34.79 over Fixed-9 at a similar average FPS, while approaching similar accuracy as Fixed-16 with latency reduced from 879 to 436 ms.

Critical strengths

The writing is generally clear and straightforward. The paper targets a specific problem with a narrow scope: to select adaptive temporal horizons for information aggregation. Developing a historical context aggregation strategy despite its perception module is practical. The proposed method brings useful computational savings with comparable accuracy. The calibration can be reused across different methods such as SAGFormer, SDGOcc, and DAOcc.

Critical weaknesses

The major issue of the submission is that the paper targets a concrete problem of accuracy-efficiency tradeoffs of the temporal context aggregation, but the necessity of such efforts is not validly justified. Tab. 4 reports integer-rounded FPS, making latency differences obscure. The unrounded latency and accuracy for fixed windows between 9 and 16 frames are unclear. It is not clear if an intermediate fixed window matches the accuracy of the adaptive one given its latency budget.

Besides the missing analysis of intermediate results between 9 and 16 frames, it is unclear if the window sizes in nearby local horizons vary significantly. Since the temporal window is selected independently for each frame, it is not clear if abrupt changes in window length leads to temporal inconsistency of occupancy predictions.

Questions for authors

The major issues are addressed in the weakness section.

1 comment

We thank the reviewer for the comment. We would like to begin by clarifying one point: our goal is to decouple the context-length selection process from the specific perceptron architecture, so that a common selection method can be adapted to arbitrary downstream occupancy predictors. Only on this basis do we then focus specifically on the accuracy–efficiency trade-off in temporal context aggregation. As for why we separate context-length selection from the specific perceptron architecture, we provide a detailed justification in the third paragraph of the Introduction in the original paper (and at the same location in the revised version).

Regarding the gap in fixed-window between frames 9 and 16 that you mentioned, we have supplemented this information in the newly submitted version. First, in Table 5 of the current newly submitted version (Table 4 in the original version), we have added the relevant information for Fixed-12 and replaced all FPS metrics in the original table with end-to-end latency. Second, in the supplementary material, we provide detailed window-by-window IoU, mIoU, end-to-end latency, and memory information from Fixed-1 to Fixed-16.

In addition, thank you very much for raising the issue of "abrupt changes in neighboring local temporal windows." In the previous version, we indeed did not adequately consider the related issues. In this version, we have conducted a simple targeted experiment (in Table 7 of the newly submitted version). In this experiment, the control group uses the window length computed from the previous frame's scene, while the window length computed for the current frame is used for the next frame. In this experiment, we find that the changes in mIoU and IoU do not appear to be particularly large, and the performance of this control group is lower than that of the experimental group. Therefore, we tend to believe that the impact of window size variations within adjacent local temporal windows is not particularly significant.

For convenience, we present all tables in Markdown format in this comment (the following is for reference only; for the specific tables, we still recommend referring to the original paper), as shown below:

GroupMethodFramesIoU↑mIoU↑Mem.↓E2E ms↓Reach16↑Harmed↓
FixedFixed-1141.4728.4727051811.0498.96
FixedFixed-3446.4030.9342332049.4375.13
FixedFixed-6749.7332.25479224630.3554.33
FixedFixed-91052.3333.35503333555.6135.62
FixedFixed-121354.2334.22597858072.4421.11
FixedFixed-161655.3834.837422879100.000.00
ControlRandom Window851.3433.19498232154.9942.38
ControlUniform 9/16952.6833.81513433759.1834.12
ControlShuffled CSE952.5533.71513436957.4435.11
AdaptiveOurs-Saturation954.9234.79511243692.644.33
OracleDisc. Suff. Oracle1255.3134.817122512100.000.00

Table: Complete fixed-window scan. HH counts historical frames and nn counts all input frames, including the current one. Models are trained independently, with one run per window. Memory is reported in MB and end-to-end latency in ms.

Hn=H+1IoU ↑mIoU ↑Memory ↓E2E latency (ms) ↓
0141.4728.472705181
1243.3629.373012191
2345.0530.203612196
3446.4030.934233204
4547.7131.454408215
5648.8231.884593229
6749.7332.254792246
7850.7132.594867265
8951.5632.924947285
91052.3033.355033335
101153.0133.665333400
111253.6933.955648485
121354.2334.225978580
131454.7034.476323678
141555.1134.666912777
151655.3834.837422879

Table: Window-length policy comparison on the full nuScenes-SurroundOcc validation set with the same frozen SAGFormer checkpoint. Both policies supply observations ending at the current frame.

Window-length policyIoU ↑mIoU ↑
Reuse preceding frame's length54.1634.45
Current CSE-selected length54.9234.79

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.