acceptodds
Under review as a conference paper at ICLR 2027

Supervising 3D Occupancy Where the Sensors Have Not Looked Yet

Abstract

Motion changes not only a driving scene, but also which parts of it can be physically verified. We turn this change in observability into supervision for 3D occupancy and a geometric verifier for driving world models. Later LiDAR sweeps often reveal static space that was hidden at an earlier query frame. Rather than collapsing these measurements into denser temporal evidence, we retain when each voxel becomes observable. This yields a query-time partition: space resolved immediately, space resolved only by later measurements (), and space never resolved in the log. is delayed physical evidence of the hidden present—not an additional model input or a future state to predict. Extending only the loss region of a matched occupancy learner from to raises F1 by on Occ3D-nuScenes; an independent five-seed retraining gives with a scene-clustered 95% confidence interval of . Standard occupancy metrics do not decline, and the gain transfers to Waymo, multiple encoders and domains, and a second lifting architecture. The same delayed evidence also gives generated futures an external geometric reference. We decode each video produced by a driving world model into 3D occupancy and check its implied static geometry at the commanded pose against a sensor-derived map. An occupancy reader trained on roughly doubles the measured separation between rollouts conditioned on logged and counterfactual actions. When choosing among four stochastic rollouts of the same counterfactual action, the resulting verifier yields about 44 more correctly located occupied voxels per frame than random selection, evaluated in a disjoint region by an independently trained reader. This sensor-grounded score is a candidate geometric reward for future world-model post-training; we do not fine-tune the generator or claim a closed-loop benefit.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D occupancy prediction, semantic scene completion, 3D scene understanding, autonomous driving, occlusion reasoning, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3KBYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes partitioning the voxel grid at a query frame by time of first observation: Robs (resolved at query time t), Rfuture (hidden at t but resolved by later ego LiDAR sweeps transformed back to the query pose), and Rnever (never resolved in the log). Later sweeps are ray-cast to produce free/occupied labels in Rfuture, turning ego motion into delayed physical supervision for geometry that was occluded at t. A matched intervention - changing only the loss mask of an occupancy learner from Robs to Robs + Rfuture - raises Rfuture F1 by +0.1156 on Occ3D-nuScenes (five-seed scene-clustered CI [+0.103, +0.122]) without degrading standard metrics, and transfers to Waymo, multiple encoders, cross-domain evaluation, and a second, depth-free lifting architecture. The paper additionally (i) profiles six released models on a coverage-precision frontier in Rfuture that standard Occ3D scoring cannot see, and (ii) uses the Rfuture-supervised occupancy model as a geometric verifier of counterfactual rollouts from the Cosmos 3 Edge driving world model, where no reference image exists, showing about 2x stronger action-conditioned separation than an observed-only reader and a measurable selection gain among stochastic rollouts.

Critical strengths
  1. Clean and well-motivated idea, executed as a controlled experiment. The core intervention changes only the loss region; architecture, schedule, seeds and labels are held fixed. The "full" arm (asserting free on Rnever) and the 90-degree rotated-label control directly address the two most obvious objections (more targets; any extra supervision). This level of experimental control is rare.

  2. Unusually honest and rigorous statistics. Matched operating points instead of argmax comparisons, scene-clustered bootstrap intervals, five-seed replication with per-seed and per-scene breakdowns, pre-specified success criteria for the second architecture (with a candid admission that "pre-specified" is not externally verifiable), and negative results reported rather than hidden (Waymo hallucination non-replication, path-proximity loss weighting doing nothing, the withdrawn angle-divergence explanation, the planner probe that cannot discriminate). Appendix G's disclosure of two near-miss statistical errors in their own code is commendable.

Critical weaknesses
  1. The headline gain is partly by construction, and the residual evidence is on a self-defined metric. Training on region X and evaluating on region X is close to tautological; the paper's real claims are (a) standard metrics do not decline, (b) the gain is not mere conservatism or blob suppression, and (c) transfer. (a) holds but the margins on standard metrics are small (matched RayIoU +0.0051); the strongest evidence is (b)-(c), which is solid but should be foregrounded even more honestly as the load-bearing result.

  2. Dynamic-object label noise is under-specified. A voxel occupied at t by a hidden object that moves away before ever being observed will be labelled free by later sweeps - a systematic error mode. Section 6.4 evaluates on objects "hidden at t and revealed after they move" (recall 0.716 to 0.856), so such voxels evidently exist in the evaluation; but the construction of their labels (which sweep, which classifier, what happens when the object is never seen while stationary) is never spelled out in the main text. This is the single most important methodological detail for a reader to trust Rfuture labels.

Questions for authors
  1. Exactly how are voxels labelled that were occupied at t by an object that moves before first being observed? Which sweep provides the occupied label, and what fraction of Rfuture labels does this affect? Is a moving-but-never-seen-while-present object labelled free?

  2. What future horizon is used to construct Rfuture by default, and how was it chosen? (Doubling the horizon moves free-space error by 0.0006, but the base horizon is not stated in the main text.)

  3. Does Rfuture supervision change semantic mIoU on the standard Occ3D protocol, even on the small models? If it cannot be tested, why not - could a released semantic checkpoint provide pseudo-labels on train?

1 comment

Thank you for the positive and careful review.

W1. Which result carries the claim. We agree. Because obs_future is trained on the region it is scored on, part of the gain is expected by construction. The evidence we rely on is that the gain is not explained by conservatism or by suppressing errors near the ego, and that it transfers. On standard metrics we claim only "no decline". The introduction now says this directly. We also add a comparison with Occ3D labels (general response, point 1), which shows that much of the gain comes from accumulating later sweeps.

W2/Q1. A voxel occupied at t by an object that leaves before it is seen. It is labelled free, because no later sweep sees it occupied. A hidden moving object gets an occupied label only if a sweep sees it before it leaves. Against nuScenes boxes, such voxels are 0.45% of R_future, and 67% of them are labelled free. Removing all voxels near annotated objects, the F1 gain is +0.128 (point 2 of the general response). The rule is now stated in Sec. 3.1.

Q2. Default horizon. 10 s: the next 20 key frames on nuScenes (plus the sweeps between them) and the next 100 frames on Waymo. It was fixed before any training. Doubling it to 20 s changes the free-space error by only 0.0006, so the never-observed region is not an artefact of the window. On nuScenes we also use a later observation only within 15 m of the sensor, because farther away a 0.4 m voxel can fall between two LiDAR beams. Both are now in Sec. 3.1.

Q3. Semantic mIoU. We could not test this during the discussion period. A semantic arm needs per-class labels for every training frame, and the semantic teacher outputs we have (from a released FlashOcc) cover only the validation split. The closest evidence is the class-stratified binary result (Sec. 6.4): with one matched threshold, the R_future gain is positive for every class group, including vehicles (+0.102) and vulnerable road users (+0.108). These are binary results, not semantic accuracy.

Reviewer_spYcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper splits each frame's voxels by when the ego LiDAR first observes them: observed now (RobsR_{obs}), observed only by a later sweep (RfutureR_{future}), or never observed (RneverR_{never}). Labels are built from raw LiDAR by ray casting, and a box-free "drift" classifier assigns the state at time tt to voxels whose state changes over the future window. RfutureR_{future} is used in two ways. First, as an evaluation region: released occupancy models are scored on space that is hidden at tt, which the standard Occ3D mask_camera protocol does not score. Second, as supervision: a small camera-only occupancy model trained on Robs∪RfutureR_{obs} \cup R_{future} instead of RobsR_{obs} improves RfutureR_{future} F1 by about +0.11 on nuScenes (scene-clustered CI [0.103, 0.122]), with the direction holding on Waymo, across encoders, and for a second lifting architecture. Finally, the trained model is used to score Cosmos world-model rollouts against accumulated real LiDAR, as a preliminary geometric verifier.

Critical strengths

The RfutureR_{future} region is a simple, annotation-free way to evaluate occupancy in space hidden at the current frame, and it reveals differences between released models that standard metrics miss. The training comparison is well controlled: the two arms differ only in the loss region, with scene-clustered intervals, multiple seeds, a pre-specified second architecture and released analysis code. I ran the supplied verification script and it reproduces the headline intervals.

Critical weaknesses
  • Occluded space is already labelled by existing datasets, and the baseline ignores this. Leaving occluded space out is a property of the mask_camera protocol, not of the labels: Occ3D ground truth is aggregated over the whole sequence, SemanticKITTI SSC scores occluded space, and training without the mask is common (FlashOcc without the mask is among the paper's own baselines). The obs arm, however, uses only the current single sweep, and obs_future does not use provenance in training, so the +0.11 gain may simply reflect that accumulated labels beat single-sweep labels. The same model should be trained on Occ3D labels (with and without the mask) and on accumulated labels without the provenance split.

  • Labels for moving objects in occluded space are unreliable and unvalidated. Voxels hidden at tt are labelled from the order of later observations (classify_drift), which fails for moving objects: a lead vehicle driving into the region occluded behind it is labelled as occupying that region at tt, a vehicle that arrives later and stops is labelled occupied at tt, and an object that leaves before being seen is labelled free. Existing datasets avoid this by placing objects with tracked boxes. The paper also says the labels "exclude dynamic actors", yet the headline metric includes these voxels, and the validity checks (static-only swept volume, simulated OPV2V) do not test them. A label error analysis against nuScenes box tracks is needed.

  • The scale of evidence is modest. The models are small (binary, ~3–4M parameters, 1/7 of the training set), the gain on standard metrics is small (RayIoU +0.007), and the world-model part rests on 18 scenes with some intervals including zero.

Questions for authors
  1. How does the same architecture perform in RfutureR_{future} when trained on Occ3D labels with and without mask_camera, or on accumulated labels without the provenance split?
  2. What is the label error rate of dynamic voxels against nuScenes box tracks, and are they included in the training loss?
  3. Are the Waymo models evaluated on frames disjoint from those used for training?
1 comment

Thank you for the detailed review, and for running the verification script.

W1/Q1. Occluded space is already labelled. You are right. Excluding occluded space comes from mask_camera, not from the labels, and Occ3D's labels are accumulated over time. Our obs_future arm is the "accumulated labels without the provenance split" that you asked about: the time of first observation is used only for scoring. The new comparison with Occ3D labels (general response, point 1) confirms that accumulation explains half to two thirds of the gain. The abstract, introduction and Sec. 3.2 now say so.

W2/Q2. Moving objects in occluded space. Thank you; this was a very useful point. The failure cases you describe are real, and yes, these voxels are in the training loss. Hidden voxels inside a moving object at t are 0.45% of R_future, and 67% of them are labelled free. Removing all voxels near annotated objects (6.3% of R_future), the F1 gain is +0.128 (+0.119 with all voxels). Details are in point 2 of the general response.

W3. Scale of evidence. We agree. The models are small and trained on 1/7 of the training frames; the gain on standard metrics is small, and we claim only that they do not decline; the world-model study uses 18 scenes. The introduction now states these limits.

Q3. Waymo. Yes. Evaluation uses 13 segments that are not used in training (51 segments).

Reviewer_sbqfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper treats later LiDAR observations as delayed supervision for geometry hidden at the query time. It partitions space by observability, extends the occupancy loss to later-observed regions, and evaluates hidden-space completion. The trained occupancy reader is also used to assess generated driving videos against sensor-derived static maps.

Critical strengths
  1. Reasonable supervision design: Later measurements provide labels for hidden geometry while avoiding unsupported labels in never-observed space.
Critical weaknesses
  1. Limited contribution: The main training change is an expanded supervision mask; its added value beyond existing future-LiDAR supervision is not sufficiently established.
  2. Poor presentation quality: Fragmented organization, incomplete method details, and uneven figure and table layout make the method and results difficult to follow.
Questions for authors
  1. Can you compare with existing supervision strategies using the same backbone and training data to isolate the added benefit of the observability-based partition?
  2. Can you demonstrate gains when adding the proposed supervision to established occupancy models under their standard training protocols?
  3. Please clarify the scene-level training, validation, and test splits for the Waymo experiments, which describe training on validation frames.
1 comment
AuthorsOpenReview ↗

Reply to Reviewer sbqf

Thank you for your review and for the clear questions.

Q1. Comparison with existing supervision. Please see point 1 of the general response. In short, Occ3D's accumulated labels give half to two thirds of our gain, and the revised paper says so.

Q2. Established models with their standard protocols. We did not test this, and we agree it is a limitation. The change is only the loss mask, so it applies to any model trained with a voxel loss; we tested two lifting designs and three image encoders. A full published model under its own protocol needs several GPU-days per run, which does not fit the discussion period. The revision states this limitation (Sec. 7).

Q3. Waymo splits. Training and evaluation segments are disjoint (51 vs. 13 segments); see point 3 of the general response.

Presentation. We are sorry the paper was hard to follow. The revision states the label rule in Sec. 3.1 and says early in the introduction what the results do and do not show.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

We thank all three reviewers for their careful and constructive reviews, and Reviewer spYc especially for running our verification script. Below we answer the three questions raised by more than one reviewer; the individual replies cover the rest. All changes in the revised PDF are in blue.

1. Is the gain just "accumulated labels beat single-sweep labels"? (sbqf Q1, spYc W1/Q1)

Largely yes. Our baseline obs uses labels from the current sweep only, and obs_future adds labels from later sweeps. The time of first observation is used only for scoring, not in the loss. We wrongly described obs as standard practice: standard Occ3D labels are also accumulated over time, and 37% of R_future already lies inside mask_camera.

To measure how much this matters, we trained the same model on Occ3D labels (same schedule, seed 0):

training labelsR_future F1geo. IoU @ mask_camera
ours, current sweep only (obs)0.5420.422
Occ3D, inside mask_camera0.6000.593
Occ3D, no mask0.6230.409
ours, current + later sweeps (obs_future)0.6610.459
  • Occ3D labels already give half to two thirds of our gain (0.542 → 0.600–0.623, vs. 0.661). The remaining difference is small, and R_future is scored with our own labels, which favours our arm. On Occ3D's own metric the Occ3D-trained model is best, as expected.
  • Training on Occ3D without the mask asserts "free" where nothing was measured, and it lowers the standard IoU below obs (0.409 vs. 0.422). Supervising only measured voxels avoids this.

So the contribution is not a new source of labels. It is (i) a way to score hidden space on its own, which standard masked evaluation does not do; (ii) evidence that supervision should stay on measured voxels; and (iii) the use of the resulting model as a geometry reader for world models. The abstract, introduction and Sec. 3.2, 4 and 5 are revised to say this, and the table is in the new Appendix P.

2. How are moving objects labelled in hidden space? (spYc W2/Q2, 3KBY W2/Q1)

Without boxes, from the order of later observations. If a voxel is seen occupied and later free, something left, so it is labelled occupied at time t. If it is seen free and later occupied, something arrived, so it is labelled free at t. Alternating observations are ignored. These voxels are in the training loss.

As the reviewers suspected, the rule makes errors: an object that leaves before it is seen is labelled free, and one that arrives later can be labelled occupied. We measured this against nuScenes box tracks on the full validation split:

  • Hidden voxels inside a moving object at t are only 0.45% of R_future, but 67% of them are labelled free.
  • 1.3% of occupied labels come from objects that arrived later.
  • Removing every voxel within 0.2 m of any annotated object (6.3% of R_future), the F1 gain is +0.128 (vs. +0.119 with all voxels).

So these labels are noisy, but they are a small part of R_future and the result does not depend on them. The rule is now stated in Sec. 3.1, and we correct the sentence saying that the labels "exclude dynamic actors".

3. Waymo splits (sbqf Q3, spYc Q3)

Training and evaluation use disjoint segments: 9,942 frames from 51 segments for training and 2,578 frames from 13 other segments for evaluation, all from the Waymo Open Dataset training split. "Trained on 2,578 Waymo validation frames" (Sec. 6.2) was a writing error, and we apologise.

A correction we found ourselves. The appendix paragraph "Not only camera placement" (App. K) is not supported. A later control, the unmoving input frame scored the same way, shows that this contrast comes from the camera pose alone. We replace the paragraph. The main-text world-model result compares both arms at the same pose and is not affected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.