acceptodds
Under review as a conference paper at ICLR 2027

Spatiotemporally Consistent Sparse World Model for 4D Occupancy Prediction and Planning in Autonomous Driving

Abstract

4D semantic occupancy forecasting provides a scene-level representation for anticipating future geometry and semantics in autonomous driving. Sparse occupancy world models enable efficient long-horizon forecasting by compressing a scene into compact latent queries and rolling them forward in time. However, existing sparse world models suffer from **spatiotemporally inconsistent semantic rollouts**. This problem arises because semantic information evolved in latent query space is not explicitly aligned with the rendered 4D occupancy structure formed by refined points and their temporal evolution. To address this limitation, we present **STC-World**, a spatiotemporally consistent sparse world model that explicitly aligns latent semantics with the rendered 4D occupancy structure. For spatial semantic alignment, we propose a **coordinate-conditioned semantic decoder** that incorporates normalized point coordinates and query-relative offsets into semantic decoding, producing location-specific corrections to point-level logits. For temporal semantic consistency, we introduce **adjacent-horizon temporal self-distillation** to constrain semantic evolution during future rollout. The distillation objective uses corrected semantics at each horizon as a stop-gradient teacher for the next horizon over inherited query indices, with class-aware consistency terms for static and dynamic categories. STC-World therefore maintains semantic correspondence throughout spatial refinement and temporal rollout while retaining a sparse latent representation. On Occ3D-nuScenes, STC-World achieves a state-of-the-art average mIoU of 13.47% over the 1–3 s forecasting horizons. It attains a 0.20 m average planning L2 error and reduces the average collision rate from 0.29% to 0.09%, corresponding to a 69.0% relative reduction, while running at 2.56 FPS with 391.2 ms mean latency. Project page: [https://stc-world.github.io/](https://stc-world.github.io/).

Primary area:
applications to robotics, autonomy, planning
Keywords:
Camera-only 4D occupancy forecasting, sparse world model, spatiotemporal consistency, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qaKzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies 4D semantic occupancy forecasting with sparse occupancy world models for autonomous driving. The authors argue that existing sparse query-based models produce spatiotemporally inconsistent semantic rollouts because latent query semantics are not explicitly aligned with the refined points that form the rendered occupancy. To address this, they propose STC-World with two components: a coordinate-conditioned semantic decoder that uses point coordinates and query-relative offsets to correct point-level semantic logits, and adjacent-horizon temporal self-distillation that uses corrected semantics at one horizon as a teacher for the next under inherited query indices, with separate consistency terms for static and dynamic classes. On Occ3D-nuScenes, STC-World reports 13.47% average mIoU, 0.20 m planning L2, and an average collision rate of 0.09%, while running at 2.56 FPS.

Critical strengths
  1. The proposed changes are lightweight and the experimental disclosure is comparatively transparent. The method adds little inference cost, and the ablations have a comprehensive coverage multi-seed variability, failure cases, and mechanism diagnostics.

  2. The diagnosis of sparse-query semantic misalignment is reasonable. Treating query-level semantics as insufficiently tied to refined point geometry is a plausible limitation of sparse occupancy world models, and Table 8 provides preliminary evidence that coordinate conditioning helps.

Critical weaknesses
  1. The occupancy gain is small and cannot be attributed to the proposed modules. Table 1 improves on the published SparseWorld numbers by 0.27 mIoU and 0.13 IoU, comparing that paper’s reported results with the authors’ own implementation. The final system is trained for 66 epochs (48+8+10), and there is no SparseWorld control continued under the same extra budget. Table 1 also omits stronger camera-based results on the same benchmark. SparseWorld-TC-Small [1] already reports 25.60 mIoU and 49.02 IoU (Du et al., 2026), well above STC-World’s 13.47 and 22.16. These numbers use future-trajectory conditioning, which Table 1 neither includes nor discusses, so the camera-based state-of-the-art claim is not supported.

  2. The ablation pattern is strongly suspicious and is left unanalyzed. In Table 4, spatial alignment and temporal distillation change average mIoU by +0.03 and +0.06, while the combination jumps by +0.29. Table 5 repeats the same pattern. The authors should analyze or draw inferences from the ablation result data. Moreover, the paper reports these numbers without explaining why nearly every partial variant returns to about 13.2 while one configuration is an outlier.

  3. The claim that spatiotemporal inconsistency is the central problem, and that STC-World solves it, is not supported by the paper’s own evidence. Fig. 1 and Fig. 3 illustrate the failure on PreWorld, so the motivating phenomenon is never measured on SparseWorld, the model this method actually modifies.

[1] Du et al. SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model. CVPR, 2026.

Questions for authors

The proposed method appears to be more of an occupancy forecasting algorithm than a world model, since action-related modeling is absent. The model is trained to predict future occupancy from historical image observations and the current ego state, and future trajectories serve as training targets rather than rollout inputs. If the authors insist on framing the task as world modeling, they should provide clear definitions and further clarify the task formulation.

7 comments
AuthorsOpenReview ↗

Response to Reviewer qaKz (1/3)

Thank you for the valuable feedback, which helped us improve the paper. We have revised the manuscript accordingly. All revisions to the manuscript are shown in blue. Below we summarize the changes, and we would appreciate it if you could let us know whether they resolve your concerns.

W1. The occupancy gain is small and cannot be attributed...

A1. We thank the reviewer for pointing out this important distinction. In this revision, we conducted an additional experiment starting from a SparseWorld checkpoint and training only the two proposed moudles. This setting still achieves a result comparable to the complete training procedure, suggesting that the improvement does not depend on extensive fine-tuning but arises from the proposed core modules.

We now cite SparseWorld-TC-Small and report additional comparisons with SparseWorld-TC-Small and I$^2$-World in Tables R1 and R2 (Tables 10 and 11 in the Supplementary Material). To make the comparison as controlled as possible, we evaluate the reproduced baselines on the same Occ3D-nuScenes validation split. All methods are evaluated on the same 4,219 samples for occupancy forecasting and the same 4,819 samples for planning. For SparseWorld-TC-Small, we use the camera-only variant without future-trajectory conditioning and evaluate both occupancy forecasting and planning. For I$^2$-World, we likewise do not provide a future trajectory as a control input and use the same evaluation samples. We retain its native precomputed LiDAR/occupancy tokens and dataset-provided transformation matrices for temporal alignment.

Both SparseWorld-TC-Small and I$^2$-World achieve lower occupancy performance under our controlled protocol than in their originally reported settings. Two changes may contribute to this difference: removing future-trajectory conditioning and adding a jointly optimized planning head with its associated losses. Despite these changes, SparseWorld-TC-Small still outperforms STC-World in occupancy forecasting, achieving an average mIoU of 21.39% and an average IoU of 43.46%. STC-World, in contrast, achieves lower planning errors, with an average L2 error of 0.20 m and an average collision rate of 0.09%, compared with 0.29 m and 0.41% for SparseWorld-TC-Small. Our clarification is that STC-World and SparseWorld-TC target different operating points: SparseWorld-TC focuses on trajectory-conditioned occupancy forecasting, whereas STC-World jointly supports camera-based occupancy prediction and downstream planning without future-trajectory control inputs.

I$^2$-World achieves an average mIoU of 10.71% and an average IoU of 21.02% under the same evaluation protocol. One possible explanation is that removing future-trajectory conditioning and jointly optimizing the planning branch affect how the forecasting model uses its precomputed LiDAR/occupancy tokens, potentially weakening their geometric and temporal alignment with future occupancy. Because this setup differs from the original I$^2$-World protocol, we report these numbers as a reference under the modified setting, not as a direct reproduction of its published results.

Table R1. Controlled occupancy comparison with additional baselines on Occ3D-nuScenes. All methods use the same occupancy samples.

MethodmIoU 1 s (%)mIoU 2 s (%)mIoU 3 s (%)mIoU Avg. (%)IoU 1 s (%)IoU 2 s (%)IoU 3 s (%)IoU Avg. (%)
SparseWorld-TC-Small23.5321.4119.2321.3946.2043.8140.3743.46
I$^2$-World14.929.877.3310.7124.8420.4717.7621.02
STC-World15.2013.4411.7713.4723.1122.2321.1522.16

Table R2. Controlled planning comparison with additional baselines on Occ3D-nuScenes. All methods use the same planning samples and evaluation protocol.

MethodL2 1 s (m)L2 2 s (m)L2 3 s (m)L2 Avg. (m)Collision 1 s (%)Collision 2 s (%)Collision 3 s (%)Collision Avg. (%)
SparseWorld-TC-Small0.230.290.350.290.330.440.460.41
I$^2$-World0.210.250.300.250.320.360.390.36
STC-World0.160.200.240.200.080.090.100.09
AuthorsOpenReview ↗

Response to Reviewer qaKz (2/3)

W2. The ablation pattern is strongly suspicious and is ...

A2. We thank the reviewer for pointing this out. We have revised the ablation discussion and included further analysis of the results. Although the partial variants follow different initialization paths, they retain largely the same learned sparse-world representation and modify only a small part of the final prediction process. Their performance therefore remains close to the common baseline level of approximately 13.2 mIoU. In Table 6, the row labeled "no alignment" means that neither coordinate-conditioned semantic alignment nor adjacent-horizon temporal self-distillation is newly enabled during the final adaptation stage. The "spatial only" row enables coordinate-conditioned semantic alignment during the final adaptation but disables adjacent-horizon temporal self-distillation. The "temporal only" row corresponds to the temporal checkpoint before the final adaptation stage. STC-World starts from the same temporal checkpoint, adds coordinate-conditioned semantic alignment, and enables adjacent-horizon temporal self-distillation during the final adaptation.

Table 7 reuses the spatial-only configuration from Table 6 as its reference. Specifically, the first row of Table 7, where neither the static KL term nor the dynamic-logit term is enabled, is the same coordinate-conditioned semantic alignment setting as the ``spatial only'' row in Table 6: coordinate-conditioned semantic alignment is enabled, while adjacent-horizon temporal self-distillation is disabled. Thus, Table 7 does not introduce a new baseline or initialization. It further adds the static KL term, the dynamic-logit term, or both to this spatial-only configuration, while keeping the query pairing and validity masks fixed. The final row therefore corresponds to the complete STC-World configuration and reproduces the 13.47 mIoU result reported in Table 6.

The similar performance of the partial variants around 13.2 mIoU indicates that each mechanism produces a conservative change when introduced alone. Coordinate-conditioned semantic alignment supplies location-specific corrections to point-level logits, while adjacent-horizon temporal self-distillation regularizes how these corrected semantics evolve across future horizons. Because the two mechanisms act on the same corrected point-level predictions, their joint optimization can produce a non-additive change in the final solution.

The additional consistency analysis shows complementary behavior. Coordinate-conditioned semantic alignment alone gives the highest boundary recall and geometry--semantic attachment, whereas the complete model produces the lowest dense semantic drift across all evaluated horizons. These results support complementary effects across different measurements.

W3. The claim that spatiotemporal inconsistency is...

A3. In the revision, we add visualizations of the ground truth, SparseWorld, and STC-World on the same scenes and forecast horizons. As show in Figure 7and Figure 8 (both in Supplementary Material), in these examples, SparseWorld produces less spatially localized occupancy and less consistent semantic layouts across future horizons, while STC-World better preserves the geometry and semantic structure of the predicted scene. These comparisons provide direct qualitative evidence on the baseline that our method modifies.

In addition, we report the dynamic- and static-class results of STC-World and the comparison methods in Table R3 (Table 5 in manuscript). The dynamic group contains bicycle, bus, car, construction vehicle, motorcycle, pedestrian, traffic cone, trailer, and truck. The static group contains drivable surface, other flat, sidewalk, terrain, manmade, and vegetation. The groups are computed as unweighted means of per-class IoUs, excluding others and barrier. STC-World improves the dynamic-class average from 10.33% to 10.76% over SparseWorld, a gain of 0.43 percentage points. The gains at 1, 2, and 3 s are 0.40, 0.44, and 0.42 points, respectively. The static-class average increases from 18.52% to 18.58%. The larger improvement for dynamic categories is consistent with the intended role of coordinate-conditioned semantic correction and temporal regularization, since these categories undergo larger changes in position and spatial support during rollout.

Table R3. Macro-averaged IoU (%) for dynamic and static class groups on Occ3D-nuScenes.

MethodDynamic 1sDynamic 2sDynamic 3sDynamic Avg.Static 1sStatic 2sStatic 3sStatic Avg.
PreWorld9.065.563.566.0618.3615.9613.7616.03
SparseWorld12.6810.148.1810.3319.6318.6617.2818.52
STC-World13.0810.588.6010.7619.6918.7217.3218.58
AuthorsOpenReview ↗

Response to Reviewer qaKz (3/3)

Q1. The proposed method appears to be more of an occupancy...

A4. We agree that STC-World does not model how a scene responds to a prescribed sequence of future ego actions. It takes historical camera observations and the observed ego state as context, then predicts future scene states through a learned latent transition. Ground-truth future occupancy and ego trajectories provide supervision for the occupancy and trajectory-related objectives during training. At inference, the occupancy rollout does not receive a ground-truth future trajectory or a candidate control sequence; future states are generated by the model's learned transition.

We use occupancy world model in the forecasting-oriented sense adopted by prior work such as PreWorld and SparseWorld. In STC-World, sparse queries represent the latent scene state, and a learned transition rolls this state forward across forecast horizons. A decoder renders each predicted state as semantic occupancy, while a planning head uses the predicted scene representation and the current ego state to generate an ego trajectory. Thus, 4D semantic occupancy forecasting is the prediction task, latent scene-state rollout is the modeling mechanism, and planning is a downstream prediction based on the forecast scene.

Reviewer_3moMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STC-World adds coordinate-conditioned semantic correction and temporal self-distillation to a sparse occupancy world model. It refines point-level semantic logits using query features, point coordinates, and query-relative offsets, then regularizes predictions across adjacent future steps with different losses for static and dynamic classes. The model supports both occupancy forecasting and motion planning.

Critical strengths

The changes are lightweight and easy to integrate into an existing sparse model. Temporal distillation adds no inference cost, while the semantic correction branch introduces little overhead. Average mIoU improves from 13.20 to 13.47 over SparseWorld, alongside better planning results. The appendix also reports five-seed results, efficiency measurements, and failure cases.

Critical weaknesses

1、The experiments omit I²-World and GenieDrive. Their published results are not directly comparable: I²-World uses ground-truth future transformations during evaluation, while GenieDrive takes occupancy and control signals as input. The authors should explain these differences and provide comparisons under matched conditions. Runtime comparisons should also clarify whether perception and occupancy decoding are included.

2、I²-World already aligns historical features using ego poses, while GenieDrive models scene changes through control conditioning and transformation supervision. What problem remains after geometric alignment, and how does the coordinate-conditioned decoder address it? More importantly, matching query and point-slot indices across time does not guarantee that they represent the same object or surface. Distillation may therefore suppress valid changes during fast motion, occlusion, or point reassignment. Comparisons with motion-based matching and targeted tests on these cases are needed.

3、Appendix A.2.1 states that both “no alignment” and “spatial only” start from a temporally distilled checkpoint, while “temporal only” receives 10 fewer training epochs. A full ablation should use the same undistilled initialization and matched training budgets. The direct consistency gains are also small: Table 12 shows an average drift reduction of only 0.056 percentage points, while geometry–semantic attachment slightly worsens. Since the planning head is further trained as well, a fixed-planner comparison would help separate the benefit of improved occupancy from additional planner training. Closed-loop evaluation would also strengthen the planning results.

[1] I^2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting. ICCV, 2025.

[2] Geniedrive: Towards physics-aware driving world model with 4d occupancy guided video generation. CVPR, 2026.

Questions for authors

see Weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3moM (1/3)

Thank you for the valuable feedback, which helped us improve the paper. We have revised the manuscript accordingly. All changes are highlighted in blue. Below, we summarize the revisions and would appreciate your feedback on whether they address your concerns.

W1. The experiments omit I$^2$-World and GenieDrive. Their published results...

A1. We now cite both I$^2$-World and GenieDrive in the related-work discussion and add controlled comparisons with I$^2$-World and SparseWorld-TC-Small in Tables R1 and R2 (Tables 10 and 11 in the Supplementary Material).

To make the comparison as controlled as possible, we evaluate the reproduced baselines on the same Occ3D-nuScenes validation split. All methods are evaluated on the same 4,219 samples for occupancy forecasting and the same 4,819 samples for planning. For SparseWorld-TC-Small, we use the camera-only variant without future-trajectory conditioning and evaluate both occupancy forecasting and planning. For I$^2$-World, we likewise do not provide a future trajectory as a control input and use the same evaluation samples. We retain its native precomputed LiDAR/occupancy tokens and dataset-provided transformation matrices for temporal alignment.

Both SparseWorld-TC-Small and I$^2$-World achieve lower occupancy performance under our controlled protocol than in their originally reported settings. Two changes may contribute to this difference: removing future-trajectory conditioning and adding a jointly optimized planning head with its associated losses. Despite these changes, SparseWorld-TC-Small still outperforms STC-World in occupancy forecasting, achieving an average mIoU of 21.39% and an average IoU of 43.46%. STC-World, in contrast, achieves lower planning errors, with an average L2 error of 0.20 m and an average collision rate of 0.09%, compared with 0.29 m and 0.41% for SparseWorld-TC-Small.

I$^2$-World achieves an average mIoU of 10.71% and an average IoU of 21.02% under the same evaluation protocol. One possible explanation is that removing future-trajectory conditioning and jointly optimizing the planning branch affect how the forecasting model uses its precomputed LiDAR/occupancy tokens, potentially weakening their geometric and temporal alignment with future occupancy. Because this setup differs from the original I$^2$-World protocol, we report these numbers as a reference under the modified setting, not as a direct reproduction of its published results.

We do not include GenieDrive as a quantitative baseline because it forecasts occupancy conditioned on an initial occupancy state and driving controls, then uses the predicted occupancy to guide multi-view video generation. These inputs and objectives differ from those of our camera-history-based forecasting and planning task, so its published results are not directly comparable under our protocol.

Table R1. Controlled occupancy comparison with additional baselines on Occ3D-nuScenes. All methods use the same occupancy samples.

MethodmIoU 1 s (%)mIoU 2 s (%)mIoU 3 s (%)mIoU Avg. (%)IoU 1 s (%)IoU 2 s (%)IoU 3 s (%)IoU Avg. (%)
SparseWorld-TC-Small23.5321.4119.2321.3946.2043.8140.3743.46
I$^2$-World14.929.877.3310.7124.8420.4717.7621.02
STC-World15.2013.4411.7713.4723.1122.2321.1522.16

Table R2. Controlled planning comparison with additional baselines on Occ3D-nuScenes. All methods use the same planning samples and evaluation protocol.

MethodL2 1 s (m)L2 2 s (m)L2 3 s (m)L2 Avg. (m)Collision 1 s (%)Collision 2 s (%)Collision 3 s (%)Collision Avg. (%)
SparseWorld-TC-Small0.230.290.350.290.330.440.460.41
I$^2$-World0.210.250.300.250.320.360.390.36
STC-World0.160.200.240.200.080.090.100.09

For the runtime comparison, we benchmarked I$^2$-World as a camera-to-occupancy pipeline comprising STCOcc camera perception, I$^2$-World tokenization, autoregressive future prediction, and dense occupancy decoding. We measured three stages separately on an A800 with a batch size of 1 and FP32 inference, using 10 warm-up iterations and 30 timed iterations per stage. The three mean stage latencies were 426.79 ms, 24.08 ms, and 282.18 ms, yielding a summed pipeline latency estimate of 733.05 ms. At the same batch size, STC-World takes 391.2 ms, including camera perception and occupancy decoding. Its latency is 341.85 ms lower, a 46.6% reduction relative to the I$^2$-World pipeline estimate. The latency ratio implies an estimated throughput of approximately 1.87×1.87\times that of the I$^2$-World pipeline at batch size 1.

AuthorsOpenReview ↗

Response to Reviewer 3moM (2/3)

W2. I$^2$-World already aligns historical features using ego poses, while GenieDrive models...

A2. We agree that ego-pose alignment and semantic alignment address different problems. Ego poses transform historical observations into a common coordinate frame, but they do not explicitly condition each point-level semantic prediction on its refined location, nor do they require semantic predictions at adjacent forecast horizons to remain compatible.

The coordinate-conditioned decoder addresses this remaining spatial issue by combining each query feature with the normalized coordinate of its refined point and its offset from the query center. It predicts a residual correction to the point-level semantic logits without modifying the recurrent rollout state. Thus, the decoder aligns the semantic prediction with the spatial support used for occupancy rendering, rather than merely aligning the input features geometrically.

We also agree that inherited query and point-slot indices do not guarantee physical object correspondence. Our method does not interpret lineage as object tracking. Instead, inherited indices provide an internal continuity prior for the sparse rollout. The chain loss is applied only to future-to-future horizon pairs, excludes the current-to-first-future transition, and uses a validity mask that removes slots that are invalid at either horizon. Newly introduced queries, disappeared queries, and invalid point slots are therefore not distilled. Moreover, static categories use a distribution-matching term, whereas dynamic categories use a weaker logit-consistency term. This reduces the risk of forcing rapidly moving objects to maintain identical semantics across horizons.

To provide category-level evidence, we report the dynamic- and static-class results of STC-World and the comparison methods in Table R3 (Table 5 in manuscript). The dynamic group contains bicycle, bus, car, construction vehicle, motorcycle, pedestrian, traffic cone, trailer, and truck. The static group contains drivable surface, other flat, sidewalk, terrain, manmade, and vegetation. The groups are computed as unweighted means of per-class IoUs, excluding others and barrier. STC-World improves the dynamic-class average from 10.33% to 10.76% over SparseWorld, a gain of 0.43 percentage points. The gains at 1, 2, and 3 s are 0.40, 0.44, and 0.42 points, respectively. The static-class average increases from 18.52% to 18.58%. The larger improvement for dynamic categories is consistent with the intended role of coordinate-conditioned semantic correction and temporal regularization, since these categories undergo larger changes in position and spatial support during rollout.

Table R3. Macro-averaged IoU (%) for dynamic and static class groups on Occ3D-nuScenes.

MethodDynamic 1sDynamic 2sDynamic 3sDynamic Avg.Static 1sStatic 2sStatic 3sStatic Avg.
PreWorld9.065.563.566.0618.3615.9613.7616.03
SparseWorld12.6810.148.1810.3319.6318.6617.2818.52
STC-World13.0810.588.6010.7619.6918.7217.3218.58
AuthorsOpenReview ↗

Response to Reviewer 3moM (3/3)

W3. Appendix A.2.1 states that both “no alignment” and...

A3. We agree with your point and have narrowed our claims accordingly. A full closed-loop evaluation requires an explicitly specified and calibrated trajectory-tracking controller, together with repeated replanning after each executed action updates the ego state. Without such calibration, tracking errors may propagate to subsequent ego states and observations, making it difficult to separate the contribution of the occupancy world model from controller behavior. Since controller calibration and iterative replanning are outside the scope of the current evaluation protocol, we instead conduct a single-stage pseudo-closed-loop evaluation to assess the compatibility of occupancy forecasts with downstream planning.

We report the single-stage reactive PDM evaluation results of STC-World and the comparison methods in Table R4 (Table 12 in the Supplementary Material). This experiment is a single-stage pseudo-closed-loop simulation rather than the complete official NAVSIM benchmark. The scenes come from 25 logs, with four consecutive samples selected from each log. SparseWorld, PreWorld, and STC-World use the same trajectory displacement conversion, single-stage reactive PDM scorer, and scene set. All three models successfully process all 100 scenes. STC-World obtains the highest average PDM score of 0.370, compared with 0.369 for SparseWorld and 0.367 for PreWorld. Under this fixed protocol, the higher overall PDM score suggests that the STC-World representation is compatible with downstream reactive planning.

Table R4. Single-stage reactive PDM evaluation on 100 fixed NAVSIM navtest scenes.

MetricSparseWorldPreWorldSTC-World
Valid / failed scenes100 / 0100 / 0100 / 0
PDM score0.3690.3670.370
No-fault collision0.8650.7200.865
Drivable area0.4300.5100.440
Driving direction0.6400.7050.645
Traffic light1.0001.0001.000
Ego progress0.5110.5380.491
TTC0.7700.6500.760
Lane keeping0.4700.6400.470
Historical comfort0.3800.3700.380
Two-frame comfort0.3600.3600.360
Reviewer_oDbjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a model and training framework for learning 3D semantic occupancy forecasting based on camera and ego state inputs. It further introduces a “coordinate-conditiioned semantic decoder” to give the model the opportunity to refine its semantic prediction. It also proposes an “adjacent-horizon temporal self-distillation” step where the model self-distills by using adjacent timesteps’ model predictions to supervise the current timesteps’ predictions. They also propose to train with the inclusion of a future ego state loss. The method is trained and evaluated on nuscenes Occ-3D dataset and shows stronger performance than the baseline methods on mIoU and is competitive on IoU.

Critical strengths

Camera-based 4D semantic occupancy forecasting is an important and challenging task making it a relevant research area to focus on. The paper raises an important failure-case of current methods where (mostly) dynamic objects are “spread out” spatially leading to unreasonable predictions. STC-world seems to handle that type of behaviour better than PreWorld in the illustrated qualitative example (Figure 3).

--

I like the idea of self-distillation part of the training, since the ground truth is not perfect, and there are many plausible futures that the model may have picked up that can act as a supervision signal.

Critical weaknesses

Paper’s writing, structure, and organization:

The paper is fairly difficult to read. It is sometimes incoherent, and uses complex sentence structures that I have difficulties unpacking and understanding. I think simplifying the language and spelling things out would help me get a better understanding. There are also places where hypotheses are phrased strongly as facts, e.g. “Semantic rollouts can become spatiotemporally inconsistent because semantics evolved in latent query space are not explicitly aligned with the rendered 4D occupancy structure.” yet are not motivated or explicitly shown. I also think the organization and structure of the paper needs to be improved. E.g. in 2.2 despite the title, planning-oriented systems are discussed. Section 3 starts with three levels of subsections before the first text. There is no chapeau with guidance of the section. Furthermore, is the mathematical notation difficult to understand. For me it is overly complex and there are missing definitions making it difficult to get an intuition of what they aim to convey. Together these things make the paper feel unpolished and would gain a lot from additional finetuning.

--

Experimental results:

The training hyperparameters seem very heavily tuned (Table 6, “Initial learning rates are 2 ×10−4 for the new decoder, 1×10−5 for existing non-backbone modules, and 1×10−6 for ResNet layer4; the chain-loss weight is 0.75.”, and “raining proceeds in three stages: 48 epochs for the base sparse world model, eight epochs with adjacent-horizon self-distillation, and 10 epochs of joint optimization after adding the coordinate-conditioned decoder.“). Since this tuning was done(?) on the validation set, which is also the set used for the final experiments it is difficult to disentangle what performance gains come from that versus the proposed method. Especially since the gains seem rather marginal. Furthermore are the results showing the aggregated metrics, but since this method aims to solve the dynamic object “smearing” it would be interesting to see a class-wise breakdown, or at least aggregating only the dynamic classe’ performance. Furthermore are the ablation tables (Tab4, Tab5) only showing the mIoU average, with small differences in performance making it difficult to get an in-depth understanding of the components’ effects. For the motion planning results, papers like Ego-MLP and AD-MLP have shown that the open-loop metrics are not reliable and it is questionable what the Table 2 performance says about the method. It would be nice to discuss this in the paper.

Questions for authors

Would it be possible to give a more fine-grained (classes/dynamic classes) performance breakdown?

Have you tried any closed-loop trajectory planning benchmarks for the system?

How sensitive is the system to hyperparameters such as learning rates, class-weighting, loss-weighting, etc?

How many timesteps are used as input to the model? How does it compare to the baselines?

How much additional training time and memory does the teacher (self-distillation) add?

7 comments
AuthorsOpenReview ↗

Response to Reviewer oDbj (1/3)

Thank you for the valuable feedback, which helped us improve the paper. We have revised the manuscript accordingly. All revisions to the manuscript are shown in blue. Below we summarize the changes, and we would appreciate it if you could let us know whether they resolve your concerns.

W1. The paper is fairly difficult to read and the organization, notation, and motivation are not sufficiently clear...

A1. We substantially revised the manuscript for clarity and structure. We simplified long and nested sentences and replaced strong factual statements with evidence-bounded formulations. We reorganized the related work by keeping the main discussion focused on 4D occupancy world models and moving the detailed planning-oriented discussion to the supplementary material. We also added a navigation paragraph at the beginning of Section 3. Finally, we unified the notation and explicitly defined all symbols.

W2. The training hyperparameters seem very heavily tuned...

A2. It should be clarified that the first stage follows the SparseWorld training protocol and learns a general initialization. The two subsequent joint-training stages constitute the training of STC-World: the first introduces adjacent-horizon self-distillation, while the second adds the coordinate-conditioned semantic decoder and jointly optimizes the specified trainable modules. In this revision, we conducted an additional experiment starting from a SparseWorld checkpoint and training only the two proposed stages. This setting still achieves a result comparable to the complete training procedure, suggesting that the improvement does not depend on extensive fine-tuning but arises from the proposed core modules.

Q1. Would it be possible to give a more fine-grained performance breakdown for individual or dynamic classes?

A3. We report the dynamic- and static-class results of STC-World and the comparison methods in Table R1 (Table 5 in manuscript). The dynamic group contains bicycle, bus, car, construction vehicle, motorcycle, pedestrian, traffic cone, trailer, and truck. The static group contains drivable surface, other flat, sidewalk, terrain, manmade, and vegetation. The groups are computed as unweighted means of per-class IoUs, excluding others and barrier. STC-World improves the dynamic-class average from 10.33% to 10.76% over SparseWorld, a gain of 0.43 percentage points. The gains at 1, 2, and 3 s are 0.40, 0.44, and 0.42 points, respectively. The static-class average increases from 18.52% to 18.58%. The larger improvement for dynamic categories is consistent with the intended role of coordinate-conditioned semantic correction and temporal regularization, since these categories undergo larger changes in position and spatial support during rollout.

Table R1. Macro-averaged IoU (%) for dynamic and static class groups on Occ3D-nuScenes.

MethodDynamic 1sDynamic 2sDynamic 3sDynamic Avg.Static 1sStatic 2sStatic 3sStatic Avg.
PreWorld9.065.563.566.0618.3615.9613.7616.03
SparseWorld12.6810.148.1810.3319.6318.6617.2818.52
STC-World13.0810.588.6010.7619.6918.7217.3218.58
AuthorsOpenReview ↗

Response to Reviewer oDbj (2/3)

Q2. Have you tried any closed-loop trajectory planning benchmarks?

A4. A full closed-loop evaluation requires an explicitly specified and calibrated trajectory-tracking controller, together with repeated replanning after each executed action updates the ego state. Without such calibration, tracking errors may propagate to subsequent ego states and observations, making it difficult to separate the contribution of the occupancy world model from controller behavior. Since controller calibration and iterative replanning are outside the scope of the current evaluation protocol, we instead conduct a single-stage pseudo-closed-loop evaluation to assess the compatibility of occupancy forecasts with downstream planning.

We report the single-stage reactive PDM evaluation results of STC-World and the comparison methods in Table R2 (Table 12 in the Supplementary Material). This experiment is a single-stage pseudo-closed-loop simulation rather than the complete official NAVSIM benchmark. The scenes come from 25 logs, with four consecutive samples selected from each log. SparseWorld, PreWorld, and STC-World use the same trajectory displacement conversion, single-stage reactive PDM scorer, and scene set. All three models successfully process all 100 scenes. STC-World obtains the highest average PDM score of 0.370, compared with 0.369 for SparseWorld and 0.367 for PreWorld. Under this fixed protocol, the higher overall PDM score suggests that the STC-World representation is compatible with downstream reactive planning.

Table R2. Single-stage reactive PDM evaluation on 100 fixed NAVSIM navtest scenes.

MetricSparseWorldPreWorldSTC-World
Valid / failed scenes100 / 0100 / 0100 / 0
PDM score0.3690.3670.370
No-fault collision0.8650.7200.865
Drivable area0.4300.5100.440
Driving direction0.6400.7050.645
Traffic light1.0001.0001.000
Ego progress0.5110.5380.491
TTC0.7700.6500.760
Lane keeping0.4700.6400.470
Historical comfort0.3800.3700.380
Two-frame comfort0.3600.3600.360

Q3. How sensitive is the system to learning rates, class weighting, and loss weighting?

A5. We report the learning-rate sensitivity results of STC-World in Table R3 (Table 16 in the Supplementary Material). The 0.5×0.5\times and 1.0×1.0\times learning-rate settings produce nearly identical average mIoU values of 13.16 and 13.15, while 2.0×2.0\times decreases the average mIoU to 13.12 and the average IoU to 22.07. The results support the stability of the method in the lower learning-rate range and show that a larger learning rate is less favorable for the final sparse world-model adaptation.

Table R3. Learning-rate sensitivity of STC-World. The 1.0×1.0\times setting denotes the default schedule.

LR scalemIoU 1s (%)mIoU 2s (%)mIoU 3s (%)mIoU Avg. (%)IoU 1s (%)IoU 2s (%)IoU 3s (%)IoU Avg. (%)
0.5×0.5\times14.8713.1311.4713.1623.1222.2521.1822.18
1.0×1.0\times14.8913.1211.4513.1523.1322.2321.1722.18
2.0×2.0\times14.8313.0711.4513.1223.0322.1221.0722.07

In addition, we report the loss-group ablation results of STC-World in Table R4 (Table 15 in the Supplementary Material). Removing future semantic or future point-refinement supervision reduces average mIoU to 3.62 and 2.01, respectively, confirming that direct future occupancy supervision is essential. Removing initial point localization, current semantic classification, current point refinement, or ego-trajectory regression produces smaller reductions to 13.21--13.23. These analyses clarify both the sensitivity to optimization scale and the contribution of each supervision group.

Table R4. Loss-group ablation of STC-World.

Removed loss groupAverage mIoU (%)
None (full objective)13.47
Initial point localization13.22
Current semantic classification13.23
Current point refinement13.23
Future semantic classification3.62
Future point refinement2.01
Ego-trajectory regression13.21
AuthorsOpenReview ↗

Response to Reviewer oDbj (3/3)

Q4. How many timesteps are used as input, and how does this compare with the baselines?

A6. We report the Camera History and Ego-State Inputs in Section A.3.1 in manuscript. STC-World uses five raw camera timestamps, consisting of the current frame and four historical frames. Each timestamp contains six surround-view images. Historical ego poses align the observed camera history, and the rollout receives only the observable current ego-state vector e0\mathbf{e}_0. Future ego poses and trajectories are used only for supervision, training-time coordinate transforms, validity masking, and evaluation; they are not rollout inputs. SparseWorld uses the same five timestamps and six cameras per timestamp. We also clarify the configuration-specific differences for PreWorld, ForecastOcc, and Drive-OccWorld, including the distinction between temporal BEV frames and additional stereo-reference images.

Q5. How much additional training time and memory does the teacher add?

A7. We report the training cost of adjacent-horizon chain distillation for STC-World in Table R5 (Table 4 in the manuscript). We added a controlled overhead measurement using the same checkpoint, one A800 80 GB GPU, batch size 6, 20 warm-up steps, and 100 measured optimizer steps. Enabling chain distillation increases the mean step time from 2.01 to 2.02 s, approximately 0.5%, and changes throughput from 2.98 to 2.97 samples/s. Peak allocated and reserved memory remains unchanged, indicating that the temporal self-distillation module introduces negligible training overhead under this configuration. Since the module is training-only, it adds no inference-time computation or parameters.

Table R5. Training overhead of adjacent-horizon chain distillation.

Training settingMean step time (s)Median (s)P95 (s)Samples/sAllocated (GB)Reserved (GB)
w/o temporal distillation2.011.992.162.9830.740.9
STC-World2.022.002.102.9730.740.9

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author–Reviewer Discussion

Thank you to all reviewers for carefully reading our work and providing constructive feedback. We have added experiments, expanded the analysis, and revised the manuscript accordingly. Due to the limited discussion period, some reviewers did not have the opportunity to further review the new results or update their scores. We kindly ask the AC to consider this additional evidence and, where possible, encourage further discussion. We summarize our response below.

1. Main concerns raised by the reviewers

The reviewers’ comments mainly focused on three aspects:

1) The paper remains somewhat difficult to follow because of long sentences, dense mathematical notation, and insufficient guidance in the organization of the related-work and methodology sections.

2) The reviewers questioned whether the proposed notion of spatiotemporal semantic inconsistency was directly measured. The original experiments mainly reported mIoU, IoU, planning L2 error, and collision rate, which do not by themselves establish that the improvements arise from better correspondence between semantics, refined geometry, and temporal evolution.

3) The reviewers suggested adding comparisons with other 4D occupancy methods and clarifying the relationship between occupancy forecasting, planning performance, and input protocols, especially for SparseWorld-TC-Small, I$^2$-World, and GenieDrive.

2. Revisions and additional experiments

1) We further split long and nested sentences, rewrote several dense passages, and reorganized the related-work discussion.

2) Although occupancy forecasting alone is narrower in scope than the joint forecasting and planning task addressed by STC-World, we included additional comparisons with SparseWorld-TC-Small and I$^2$-World. These results highlight STC-World’s balanced performance across occupancy forecasting and motion planning.

3) We added dynamic- and static-class IoU breakdowns. The dynamic-class improvement is consistent across all three forecast horizons, indicating that the gain is more pronounced for objects whose spatial support changes over time.

4) We added a single-stage reactive PDM evaluation on 100 fixed NAVSIM scenes. This experiment evaluates the compatibility between occupancy forecasts and downstream reactive planning.

5) We added per-horizon component ablations for coordinate-conditioned semantic alignment, adjacent-horizon temporal self-distillation, and the static and dynamic consistency terms.

6) We added visual comparisons between SparseWorld and STC-World on the same scenes and forecast horizons. The results show that some dynamic or local occupancy structures in SparseWorld become increasingly diffuse over time, whereas STC-World better preserves their local geometry and semantic support.

7) We added a learning-rate sensitivity study. The results indicate that the method is stable in the lower learning-rate range, while an excessively large learning rate degrades the final adaptation.

8) We added a training-cost analysis for adjacent-horizon self-distillation. Enabling chain distillation increases the mean step time from 2.01 s to 2.02 s, corresponding to approximately 0.5%, while peak allocated and reserved memory remain unchanged.

3. Significance of the work

The central contribution of this work is to identify a specific representational issue in sparse world models: although semantic states can be rolled forward in query space, they are not explicitly constrained to remain aligned with the refined points used for rendering and with their temporal evolution.

Compared with OccWorld[1] and PreWorld[2], which establish future occupancy prediction through latent or tokenized scene rollouts, and SparseWorld[3], which improves rollout efficiency with dynamic sparse queries, STC-World introduces explicit point-level spatial semantic alignment and adjacent-horizon temporal regularization. These mechanisms are complementary to the underlying rollout architecture and directly address the correspondence between latent semantics, refined geometry, and future scene evolution.

[1]Occworld: Learning a 3d occupancy world model for autonomous driving, ECCV2024.

[2]Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving, ICLR 2025.

[3]SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queries, AAAI 2026.

Taken together, the revised manuscript provides a more complete and carefully scoped evaluation of STC-World. We respectfully ask the AC to consider its methodological contribution to semantic alignment in sparse camera-based world models, the direct consistency and dynamic-class evidence, and its balanced occupancy--planning performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.