acceptodds
Under review as a conference paper at ICLR 2027

OccSim: Multi-kilometer Driving Simulation with Long-horizon Occupancy World Models

12 views

Abstract

Data-driven driving simulation relies on pre-recorded logs or spatial priors such as HD maps, so the environments it can produce are capped by the scale of existing annotated datasets. To break this bottleneck, we present OccSim, the first 3D occupancy world model driven simulator that requires neither HD maps nor log-derived layouts. OccSim obviates the requirement for long continuous logs or HD maps; it is conditioned only on a single initial frame and a sequence of future ego-actions. It can stably generate over 3000 continuous frames, an 80x improvement in stable generation length over previous SoTA occupancy world models, and fuses them into maps spanning over 9 kilometers. Further qualitative stress tests show that it can roll out beyond 110K frames. OccSim couples W-DiT, a static occupancy world model that builds rigid transformations into its architecture for ultra-long-horizon rollout, with a Layout Generator that populates the synthesized topology with reactive agents. With these designs, OccSim can synthesize massive, diverse simulation streams. Extensive experiments demonstrate its downstream utility: data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance on unseen data, outperforming previous asset-based simulator by 11%. When the collected data is scaled up to 5x its original size, zero-shot performance increases to 74%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Occupancy World Model, Autonomous Driving, Generative model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_JjfjOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

OccSim generates large occupancy-based driving environments from an initial static frame and future ego actions. Its W-DiT backbone combines rigid latent warping, spatial conditioning, and an SNR-weighted occupancy loss for long rollouts. Generated frames are fused into a persistent map, populated by a learned layout generator, and simulated with a replaceable controller. The paper reports quantitative generation up to approximately 10 km and downstream sim-to-real forecasting improvements.

Critical strengths

(+) The problem is significant, and the separation between offline environment generation and online agent simulation is sensible.

(+) The long-horizon results are substantial. At 10 km, Table 1 reports 2D-FID of 47.13 and an overall LCC ratio of 92.90%, compared with 148.29 and 11.70% for the reproduced SemCity baseline. The paper jointly evaluates fidelity, diversity, and connectivity and explicitly distinguishes quantitative results from the much longer qualitative stress test.

(+) Downstream validation provides evidence beyond generative metrics.

Critical weaknesses

(-) The current metrics do not adequately capture persistent global consistency. Marginal FID and local diversity can remain favorable even when spatial inconsistencies exist. Among the reported metrics, LCC measures connectivity rather than the geometric correctness of revisited regions. Have the authors considered quantifying revisit consistency, loop-closure error, and road validity before and after fusion to provide more direct evidence for the global consistency claim?

(-) The evidence for dynamic simulation remains more limited than that for environment generation. Table 2 identifies the controller and initialization as major error sources, and Appendix 7.2 reports closed-loop collision statistics. However, downstream data collection limits each scene to six agents and retains only collision-free clips, while the learned policy is evaluated only in an open-loop setting. Could the authors consider adding closed-loop evaluations of the learned policy, particularly in denser traffic and more interactive scenarios?

Questions for authors
  1. How consistent are overlapping or revisited regions before map fusion, and how much does the fusion procedure improve or alter them?

  2. What fraction of generated simulation clips is discarded by collision filtering, and how does filtering change the behavior distribution?

  3. Do downstream gains persist under matched optimizer steps and when training and evaluation are repeated across independently generated maps?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Jjfj (1/2)

We thank Reviewer Jjfj for the constructive review, and for recognizing the significance of the problem and the simulation pipeline we build. We have addressed the weaknesses raised regarding the fusion stage and the downstream simulation with new experiments and revisions in the paper. Since the weaknesses and questions overlap, we reorganize them into the following points and answer each in turn:

  • How well do overlapping and revisited regions agree before fusion? (see General Response)

1. The effect of fusion on occupancy fidelity

How much does the fusion procedure improve or alter the quality of overlapping regions?

We answer this in two parts.

Alteration. Applying the same fusion procedure to GT keyframes from the val set and to W-DiT rollouts, we compare each fused map with its raw keyframes:

DataKeyframesIoUmIoUAddedRemoved
GT44757.0654.0825.3%17.7%
W-DiT rollout1,10351.1352.6233.5%15.4%

These changes reflect fusing view-limited per-frame occupancy rather than correcting generation errors: fusion alters GT keyframes to a comparable extent, and removes even fewer voxels from W-DiT keyframes than from GT ones.

Improvement. Following the straight-trajectory setting in the paper, we roll out W-DiT for 200 frames and compare the map obtained by directly stitching the raw keyframes, which do not overlap, with the map after fusion:

2D-FID@10 ↓2D-FID@35 ↓2D-FID@100 ↓2D-FID@200 ↓LCC ↑
GT keyframes (w/o fusion)26.9925.61––97.6%
GT + fusion38.5836.50––100%
W-DiT keyframes (w/o fusion, Fig. 4)42.8551.2048.6746.4074.1%
W-DiT + fusion56.3774.8479.4181.0792.5%

In terms of 2D-FID, fusion does not improve fidelity, since the reference samples are per-frame labels before densification; the increase reflects the loss of sensor-like sparsity rather than degraded content, as it occurs for GT as well. Fusion does, however, improve LCC, since the same densification restores connectivity: sparse far-range observations leave holes that break directly stitched keyframes into disconnected fragments, and fusion fills them with votes from the non-keyframes.

2. Further analysis of clips discarded due to collisions

What fraction of generated simulation clips is discarded by collision filtering, and how does filtering change the behavior distribution?

This is reported in Tab. 7 (Sec. 7.2). For downstream data collection we use at most 6 agents and 10 s clips, for which 38% of the clips contain a collision and are discarded. These collisions mainly stem from the rule-based IDM controller and layout initialization rather than the generated maps, consistent with Tab. 2. To measure how filtering changes the behavior distribution, we compare it with two alternatives: truncating each clip at its first collision, and keeping the full 10 s clip regardless of collisions.

ModeDiscarded clipsLin. Spd.Ang. Spd.Acc.Near. Dist.
Discard collided clips (Tab. 2, Ours)38%8.760.3518.61.40
(1) Truncate at collision38%8.710.3818.91.47
(2) Keep full 10 s clips0%12.760.4518.82.39

JSD (×10⁻²) to real behavior, following the protocol of Tab. 2.

Keeping full clips introduces a simulation artifact: after a collision, the IDM followers stop behind the crashed vehicles, so traffic jams completely and the clip is dominated by near-zero speeds and abnormally small gaps. Truncating at collision avoids the jam but keeps the pre-collision behavior, and yields a distribution close to that of discarding. Discarding collided clips therefore removes an artifact of the rule-based controller rather than selecting favorable behavior.

AuthorsOpenReview ↗

Response to Reviewer Jjfj (2/2)

3. Sensitivity of downstream gains to optimizer steps and generated maps

Do downstream gains persist under matched optimizer steps and when training and evaluation are repeated across independently generated maps?

Thank you for raising this. We address the two factors separately.

Matched optimizer steps. To separate the effect of data from that of training length, we retrain OccWorld on the 1× OccSim data for 5× the optimizer steps, matching the 5× setting in Tab. 3:

Training dataOptimizer stepsIoU (1 s / 2 s / 3 s)mIoU (1 s / 2 s / 3 s)
OccSim 1×N (Tab. 3)20.53 / 15.48 / 13.2413.51 / 9.56 / 7.61
OccSim 1×5N (new)21.08 / 15.31 / 13.4114.01 / 9.60 / 7.54
OccSim 5×5N (Tab. 3)23.87 / 16.92 / 13.4719.54 / 12.31 / 9.08

Training longer on the same data brings no meaningful gain: all changes are within 0.6 points, and some horizons slightly decrease, indicating that the 1× model has already converged. Under matched optimizer steps, the 5× data still improves mIoU by 5.53 / 2.71 / 1.54 at 1 s / 2 s / 3 s and IoU by 2.79 / 1.61 at 1 s / 2 s, while IoU at 3 s is on par. The gains from scaling OccSim data therefore come from the additional data rather than from more optimization.

Independently generated maps. To verify that the downstream gains do not depend on a particular generated map, we independently generate two additional maps with different initial frames and random seeds, keeping the trajectory length and data collection protocol unchanged, and train OccWorld on each under the 1× setting:

Training dataMetricMap 1 (Tab. 3)Map 2 (new)Map 3 (new)Mean ± std
OccSim 1×IoU20.53 / 15.48 / 13.2419.99 / 14.94 / 13.1521.01 / 15.67 / 13.0320.51 ± 0.51 / 15.36 ± 0.38 / 13.14 ± 0.11
mIoU13.51 / 9.56 / 7.6114.20 / 9.15 / 7.5913.59 / 9.61 / 7.3313.77 ± 0.38 / 9.44 ± 0.25 / 7.51 ± 0.16
CARLA (Uni-C), all mapsIoU–––18.31 / 14.09 / 12.03
mIoU–––11.79 / 8.35 / 6.75

Entries are at 1 s / 2 s / 3 s; std is the sample standard deviation over the three maps.

Note that the CARLA data is already collected over all available CARLA maps, so the per-map columns are left blank for it, whereas each OccSim run uses a single generated map. The results are highly consistent across maps, and the map used in Tab. 3 lies close to the mean. Each of the three maps outperforms CARLA at every horizon on both IoU and mIoU, with a gap to CARLA 3–10× larger than the standard deviation across maps.

4. Simulation with higher agent density

How does the simulator behave in denser traffic, and can the learned policy be evaluated in closed loop?

Tab. 7 (Sec. 7.2) reports the effect of traffic density from 4 to 7 agents: the collision rate at 10 s rises from 0.12 to 0.41, and at 60 s from 0.49 to 0.79. This increase stems from the rule-based controller rather than the generated environment: each agent drives toward a randomly selected exit, and the 2D-IDM has no intersection protocol such as right-of-way or signal phases, so agents increasingly block each other as density grows. Dense, highly interactive traffic therefore calls for a stronger controller, which OccSim is designed to accommodate: the controller is a replaceable module, and since the environment is pre-built and the ego vehicle is part of the agent set (Alg. 6, l.6), a learned policy can replace IDM for the ego or for other agents without modifying environment generation. We agree that closed-loop evaluation of learned policies in dense traffic is an important next step, and have added this discussion to the limitations of IDM-based agent behavior simulation in Sec. 10.

We hope these clarifications, revisions and new experiments address your concerns. If so, we would be grateful if you would consider updating your assessment. We are happy to clarify any remaining questions during the discussion period!

Dear Reviewer Jjfj,

As the discussion period ends tomorrow, we would like to kindly follow up on our response. Following your suggestions, we have added new experiments for each of your questions:

  • Revisit and loop-closure agreement (General Response): pairwise agreement of raw frames before fusion against a GT reference, showing that disagreement does not accumulate along the rollout, and the agreement at the loop-closure seam, which exceeds that of GT frame pairs with comparable overlap.
  • Effect of fusion (point 1): fusion alters GT and generated keyframes to a comparable extent, and raises LCC from 74.1% to 92.5% for W-DiT.
  • Collision filtering (point 2): the fraction of discarded clips and its effect on the behavior distribution, showing that filtering removes controller artifacts rather than selecting favorable behavior.
  • Robustness of downstream gains (point 3): under matched optimizer steps and across three independently generated maps, the gains over CARLA persist with low variance.

For closed-loop evaluation of learned policies in dense traffic, we have clarified the scope of the current controller and added it to the limitations. We would greatly appreciate it if you could let us know whether these results address your concerns. If they do, we would be grateful if you would consider updating your assessment.

Thank you again for your time and constructive feedback.

Dear Reviewer Jjfj,

We have now incorporated all changes into the revised PDF, marked in blue. In particular, the agreement of overlapping and revisited regions before fusion is reported in Sec. 4.2 (Tab. R1), the effect of fusion in Sec. 7.10 (Tab. R5–R6), and the robustness of downstream gains in Sec. 7.9 (Tab. R3–R4); a summary of all revisions is given in our comment "Summary of revisions".

We would sincerely appreciate it if you could take a look at the revision and, if your concerns have been addressed, consider updating your assessment. Thank you again for your time and constructive feedback.

Reviewer_LrbKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a novel driving simulator based on W-DiT and layout generator. The W-DiT incorporates local transformation with forward mapping, while it generates static road information with SNR-scaled loss. The layout generator rolls out the agents which are ampped from a 2D heatmap. Overall, the biggest contribution is that this work extends the prediction scale to a large extent. Long-horizon generation (up to 110K frames) is crucial in data curation and resimulation in autonomous driving.

Critical strengths
  1. The technique novelty is strong and reasonable. The authors combine many crucial components in generative modeling and autonomous driving fields. Together, these contributions enable the network to roll out long-horizon futures.

  2. The disentangled static and dynamic make sense. The overall static roadmaps are first generated frame-by-frame and then concatenated to form a big map. And then the dynamic agents are also generated without predefined logs.

  3. Authors provide pseudo-code in the main novelty design, which facilitates reproducing the results.

  4. From the video link, the generated scenes are physically plausible and also the agent motions are consistent.

  5. A novel log-free evaluation protocol is proposed, combining realism, diversity, and downstream transfer.

Overall, this work will have great influence in the real-world AR simulation with extreme long-horizon rollouts.

Critical weaknesses
  1. The paper lacks of detailed descriptions in some method parts. For example, how to do gravity alignment and why; how to conduct threshold voting; how the controller is designed for foreground motion, etc. More explanations are better for readers to understand the details.

  2. The evaluation metrics mainly focus on the diversity and realism. However, in the real-world application, efficiency is also important. More analysis of the break-down efficiency is needed.

  3. In Table 1, the comparison methods seem to be limited. There are more recent occupancy-based world models and integrating them in comparison will make this work even more solid.

Questions for authors

Overall, this paper proposed a novel and practical occupancy world model for AR simulation. The contributions are sufficient. The main concern is about lack of detailed explanations on some crucial components, which makes readers hard to follow.

Also, there is another work (VectorWorld, ICML 26) also studying large-scale AR simulation but based on vector graph. The authors should mention it as concurrent similar work.

5 comments
AuthorsOpenReview ↗

Response to Reviewer LrbK (1/2)

We thank Reviewer LrbK for the encouraging assessment of both the technical novelty and the comprehensiveness of our evaluation. Due to the page limitation in the initial submission, we agree that several components were described too briefly. Using the additional page allowed during discussion, we will revise them in the final version of PDF. Below, we address each of your comments in turn.

1. More details about component designs:

The paper lacks of detailed descriptions in some method parts. For example, how to do gravity alignment and why; how to conduct threshold voting; how the controller is designed for foreground motion, etc. More explanations are better for readers to understand the details.

a. Why use gravity alignment and how is it implemented?

We first note that this step appears as "column sinking" in Alg. 2 (Sec. 9.1); we have unified the naming in the revision.

As to why we perform gravity alignment: W-DiT assumes planar ego motion (Sec. 3.1, l.215–216) for simplicity, so that the poses contain no pitch, roll or vertical displacement transformations. However, the real trajectory we use as user-steered input, can, in general, contain such transformations. These transformations could result in the generated occupancy being unconstrained to level ground. While such transformations may be useful to retain in some cases (e.g., driving up and down hills, or rolling on heavily banked turns), , they introduce more complexity for generative modeling, agent control, and lane graph extraction. Thus, gravity alignment is used to enforce planar motion assumptions for simplicity.

As to how we perform gravity alignment: After each keyframe is written into the global map, every (x, y) column is shifted vertically so that its lowest ground voxel lies on z = 0 (Alg. 2 l.13–14), and the remaining empty cells on z = 0 are filled by the mode of their 3×3 neighborhood (Alg. 2 l.15). Since the shift is applied per column, above-ground structures such as buildings, vegetation, and poles keep their heights relative to the ground.

b. How is threshold voting implemented?

In the submission, this step is described in Sec. 9.1 (l.1565) and Phase 3 of Alg. 2 (Non-Keyframe Voting Inpainting). Its purpose is to exploit the redundant non-keyframes to fill regions left unobserved or sparse after the keyframe pass. Specifically, each non-keyframe is warped into the global coordinate and votes only for voxels still unassigned after the keyframe pass, so keyframe content is never overwritten (Alg. 2, l.17–20). A voxel takes the most-voted category if it receives at least τ\tau votes and otherwise remains empty (l.21). Here we use τ=3\tau = 3.

c. How is the controller designed for foreground motion?

The controller is described in Sec. 9.4 and Alg. 6. Each agent drives along an B-Spline smoothed A* route on the extracted lane graph toward a randomly selected end of branch road (Alg. 5), and interacts with others through following rules:

  • Car-following. An agent treats as its leader the nearest vehicle that lies within its 60° forward cone and within dlatd_{\text{lat}} of its planned path (Alg. 6, l.20–21), and its acceleration is given by IDM from the gap ss and relative speed Δv\Delta v to this leader (l.22, l.29). Each agent samples its own desired speed vdes∼N(μv,σv)v_{\text{des}} \sim \mathcal{N}(\mu_v, \sigma_v) (Alg. 5, l.4), so faster agents catch up with slower ones.
  • Lane changing. If the gap falls below dlcd_{\text{lc}} while approaching a slower leader (Δv>0\Delta v > 0) or a head-on vehicle (ha⋅hlead<−0.5h_a \cdot h_{\text{lead}} < -0.5), the agent switches to an adjacent parallel lane via a Bézier curve and re-plans with A* from there (l.23–28).
  • Rolling horizon. Every 40 m of ego motion, agents outside the ego's field of view are removed and new ones are spawned by the layout generator ahead of and behind the ego (l.11–16), keeping traffic density around the ego stable over arbitrarily long drives.
  • Ego in the loop. The ego vehicle is included in the agent set (l.6) and follows the same rules, so it interacts with surrounding traffic in closed loop.
AuthorsOpenReview ↗

Response to Reviewer LrbK (2/2)

2. More analysis of the break-down efficiency is needed

The evaluation metrics mainly focus on the diversity and realism. However, in the real-world application, efficiency is also important. More analysis of the break-down efficiency is needed.

This analysis is provided in Sec. 7.1 (Tab. 5 and Tab. 6); we have added a pointer to it in the main text. In summary:

World model. W-DiT (181M parameters) runs at 2.58 FPS on a single RTX 4090, comparable to occupancy world models such as DOME (5.48 FPS) and COME (2.33 FPS), and over 100× faster than InfiniCubes (0.02 FPS) (Tab. 5).

Environment construction. Building one environment from an initial frame and a 1000-step action sequence (encoding, W-DiT rollout, decoding and lane extraction) takes about 656 s, i.e. roughly 3.6 min per km at 3 m per step. This cost is paid once per environment, not per simulated frame (Tab. 6).

Simulation. Once the map is built, each simulation step costs only 7.1 ms for IDM planning and rendering, plus a layout recomputation (92.68 ms) every 40 m of ego motion.

This split reflects our two-loop design: the expensive generation loop is amortized offline, while the online simulation loop runs at the cost of a rule-based controller. All numbers use float32 without compilation or other acceleration and are thus an upper bound.

3. More recent occupancy-based world models comparison

Comparison methods seem to be limited. There are more recent occupancy-based world models and integrating them in comparison will make this work even more solid.

We thank reviewer for pointing this out. We would like to clarify that Table 1 here compares our method with other non-world-model-based approaches. If you are referring to world-model-based methods, our detailed comparison is presented in Figure 4. As a supplement, we show the experiments compare with more latest methods here, which will be added to the paper. We retrained the following latest work on the static occupancy and use straight trajectory to rollout 200 frames, FID of rollout frames are reported below.

ModelVenueFID @5 framesFID @20 framesFID @50 framesFID @100 framesFID @200 frames
GenieDriveCVPR 202644.0576.75111.62125.72134.92
COME (Fig 4)NeurIPS 202532.8281.25105.62107.82108.53
Ours (Fig 4)-36.6442.8551.2148.6746.40

The results once again confirmed our hypothesis (l. 86–87): the accumulation of errors resulting from prolonged rollout is inherent to the Spatiotemporal Transformer architecture; even in the latest work, as long as this design is retained, rollout collapse is inevitable.

4. More comprehensive related work.

There is another work (VectorWorld, ICML 26) also studying large-scale AR simulation but based on vector graph.

We thank the reviewer for pointing out this concurrent work, and will add it to Sec. 2.3 of the revised paper as follows:

Each of these still draws its environment from an external source: recorded log, HD map, or layout to be outpainted. Concurrently, VectorWorld (Jiang et al., 2026) takes a step in this direction by outpainting lane-agent vector graphs at scale, but still learns its road representation from HD-map annotations. Therefore, the environments available for simulation remain bounded by annotation effort rather than by generation.

We hope these clarifications, revisions and new experiments address your concerns. If so, we would be grateful if you would consider updating your assessment. We are happy to clarify any remaining questions during the discussion period!

Dear Reviewer LrbK,

As the discussion period ends tomorrow, we would like to kindly follow up on our response. In your review, you noted that the contributions are sufficient and that the main concern is the lack of detailed explanations of some crucial components. We have addressed this point by point, describing gravity alignment, threshold voting and the agent controller in detail, together with the efficiency breakdown, a new comparison with the recent GenieDrive (CVPR 2026), and the discussion of VectorWorld.

We would greatly appreciate it if you could let us know whether these clarifications resolve your concerns, or whether any point remains unclear. If they do, we would be grateful if you would consider updating your assessment.

Thank you again for your time and constructive feedback.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General response

General Response

We first address a question raised by two reviewers: how well overlapping and revisited regions agree before fusion.

Reviewer Jjfj: How consistent are overlapping or revisited regions before map fusion?

Reviewer 6Czd: Does a revisit consult the global map or only the previous latent? An overlap-disagreement measurement before fusion would clarify the consistency claim.

Agreement of overlapping regions. To quantify the agreement of overlapping regions before fusion, we measure the agreement between pairs of raw frames separated by a displacement Δ, on 150 nuScenes val scenes with 10 static classes. Both frames are placed in a common frame using yaw-only transforms with nearest-neighbor voxel sampling; IoU measures occupied versus empty, and mIoU averages over the 9 static semantic classes. Each frame spans 80 m, so a displacement Δ leaves an overlap of (80 − Δ) m; since each step moves about 3 m, Δ = 60 m corresponds to 20 autoregressive iterations with a remaining overlap of 20 m.

Δ (overlap)GTW-DiT (same frame pairs as GT)W-DiT (pairs incl. extrapolated frames)
5 m (75 m)51.96 / 47.4838.91 / 32.5941.34 / 35.07
10 m (70 m)43.86 / 38.5130.33 / 23.9931.73 / 25.91
20 m (60 m)36.98 / 30.8422.69 / 16.9022.69 / 17.71
40 m (40 m)31.17 / 24.5515.89 / 10.7615.02 / 10.42
60 m (20 m)27.70 / 19.3512.30 / 7.1311.03 / 6.65

Entries are IoU / mIoU.

Three observations follow. First, even per-frame GT pairs agree only partially, since Occ3D labels carry view-dependent visibility and occlusion. Second, W-DiT, trained on such labels, is closest to GT at short displacements, and the gap widens with Δ, as expected since W-DiT re-synthesizes the complete frame conditioned on the warped latent. Third, comparing the last two columns, agreement is essentially unchanged when the pairs include extrapolated frames, so the disagreement does not accumulate along the rollout.

Revisited regions. For revisiting, only the previous latent is used. For a closed loop, we store the latent of the first frame and reuse it when the trajectory returns to the closing point. Specifically, when the trajectory is 30 m away from the starting point, a single frame bridges the remaining gap: of its 80 m extent, the rear 25 m overlaps the preceding frame and the front 25 m overlaps the stored first frame, leaving the central 30 m to be generated. The conditioning of W-DiT for this frame is thus composed of the rear part of the warped latent from its preceding step and the front part of the warped stored first-frame latent. W-DiT therefore synthesizes the closing frame with context from both ends of the loop, stitching the end to the beginning rather than generating it from one side only.

OverlapComparisonW-DiTGT
40 mFrame pairs (Δ = 40 m)15.89 / 10.7631.17 / 24.55
25 mLoop-closure seam vs. stored first frame35.62 / 31.79–
20 mFrame pairs (Δ = 60 m)12.30 / 7.1327.70 / 19.35

Entries are IoU / mIoU on the 150 nuScenes val scenes.

On the 150 val scenes, the 25 m seam region agrees with the stored first frame at 35.62 / 31.79 IoU / mIoU, well above raw W-DiT frame pairs with comparable overlap and even above GT pairs. Since the first-frame latent is given explicitly as conditioning for the closing frame, its content is largely carried over across the seam rather than re-synthesized independently.

The global map is not consulted during generation; it is assembled afterwards by the fusion stage (Sec. 3.2), which also reconciles any residual disagreement at the seam. We will add this description to Sec. 3.2.

AuthorsOpenReview ↗

Response to Reviewer 6Czd (1/2)

We thank Reviewer 6Czd for the careful and detailed reading, and for recognizing that our long-horizon results are supported by complementary evaluations and that the generated environments have demonstrated downstream utility. We also thank the reviewer for pointing out several ambiguous claims and settings. We have clarified each of them in the revised paper and discuss these clarifications below. Since the weaknesses and questions overlap, we reorganize them into the following points and answer each in turn:

  • How well do overlapping and revisited regions agree before fusion? (see General Response)

1. Supervised region of the perception loss and mapping details

Which region does the Eq. (3) cross-entropy supervise, and how is its mask mapped to occupancy resolution?

Thank you for catching our typo; the definition of MmaskM_{\text{mask}} below Eq. (3) contained a notation error. It should read Mmask=1−(Mvis⊙Mrand)M_{\text{mask}} = 1 - (M_{\text{vis}} \odot M_{\text{rand}}), i.e., the region newly revealed after warping together with the randomly dropped latents, so the retained conditioning latents receive no perception loss. Since the latent zt+1∈R50×50×16z_{t+1} \in \mathbb{R}^{50 \times 50 \times 16} has no height axis, MmaskM_{\text{mask}} is mapped to the resolution of Ot+1O_{t+1} by nearest-neighbor upsampling by 4× in x and y (50×50 → 200×200) and broadcasting along the height axis. We use λ=2\lambda = 2. Using the decoded ground-truth latent D(zt+1)\mathcal{D}(z_{t+1}) instead of Ot+1O_{t+1} as the target yields nearly identical results. We have corrected the definition and added the mapping in Sec. 3.1.

2. Preservation of known content at inference

At inference, is known content explicitly preserved or used only as conditioning?

No. Known content is used only as conditioning and is not copied into the output. At each step, W-DiT synthesizes the full frame, and its preservation of the visible region comes from the architecture itself: the dual-head injection modulates every spatial token by its own warped condition (Sec. 3.1), and the velocity term in Eq. (3) supervises the full frame. Sec. 7.5 is therefore correct, while the sentence in Sec. 3.1 (l.215) was imprecise. We will revise it to: "…warping the latent so that the network mainly needs to synthesize the newly revealed region, with the warped content serving as conditioning." Since the visible region is re-synthesized rather than copied, repeated observations of the same location can differ slightly; this is why the fusion stage (Sec. 3.2) is needed, and we quantify this disagreement before fusion in the General Response.

AuthorsOpenReview ↗

Response to Reviewer 6Czd (2/2)

3. Description in Sec. 3.1 vs. Sec. 7.5, and overlap agreement before fusion

Inconsistencies in the description: Section 3.1 says only newly revealed regions are generated, whereas Section 7.5 says the complete frame is re-synthesized.

We thank the reviewer for pointing out this inconsistency. The statement in Sec. 3.1 (l.215–216) that "only the newly revealed region is generated" describes the ideal case in which the network fully reproduces the warped content, i.e., the upper bound of the method. In practice, as stated in Sec. 7.5, W-DiT re-synthesizes the complete frame conditioned on the warped latent. Unlike prior occupancy world models, however, this re-synthesis remains stable over long horizons. We will revise Sec. 3.1 accordingly (see point 2), and quantify the agreement of overlapping regions before fusion in the General Response.

4. Shared settings in Tab. 4 and the isolated effect of dual-head injection

Which VAE/readout and conditioning settings are shared across Table 4 and the static retrained COME comparison? What can these controls establish about dual-head injection independently of the complete recipe?

All rows of Tab. 4 share the same sphere-readout VAE, training data and conditioning information, and differ only in the marked components. The static-retrained COME in Fig. 4, in contrast, keeps its original VAE and training setup, and thus serves as a system-level comparison rather than a component ablation.

In Tab. 4, "W-DiT" denotes the architecture with dual-head injection, so removing dual-head injection necessarily falls back to the classical DOME-style spatiotemporal DiT. The architecture comparison is therefore given by rows 2 and 5: under the same VAE, perception loss and SNR scaling, replacing W-DiT with the spatiotemporal DiT raises the 500-frame FID from 744.71 to 1607.97. Notably, the spatiotemporal DiT is given 4 history frames, whereas W-DiT conditions on only 1 warped previous frame; W-DiT thus achieves stable static occupancy generation with less historical information.

5. Post-processing in the LCC comparison

Specify the post-processing applied to each method in the LCC comparison.

All methods for which LCC is reported (SemCity and OccSim) use the same fusion procedure described in Sec. 3.2 (Alg. 2), so the comparison is fair. UniScene produces only single-frame outputs, so LCC is not reported for it. We have added this clarification to Sec. 4.2.

We hope these clarifications, revisions and new experiments address your concerns. If so, we would be grateful if you would consider updating your assessment. We are happy to clarify any remaining questions during the discussion period!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.