- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
OccSim generates large occupancy-based driving environments from an initial static frame and future ego actions. Its W-DiT backbone combines rigid latent warping, spatial conditioning, and an SNR-weighted occupancy loss for long rollouts. Generated frames are fused into a persistent map, populated by a learned layout generator, and simulated with a replaceable controller. The paper reports quantitative generation up to approximately 10 km and downstream sim-to-real forecasting improvements.
Critical strengths
(+) The problem is significant, and the separation between offline environment generation and online agent simulation is sensible.
(+) The long-horizon results are substantial. At 10 km, Table 1 reports 2D-FID of 47.13 and an overall LCC ratio of 92.90%, compared with 148.29 and 11.70% for the reproduced SemCity baseline. The paper jointly evaluates fidelity, diversity, and connectivity and explicitly distinguishes quantitative results from the much longer qualitative stress test.
(+) Downstream validation provides evidence beyond generative metrics.
Critical weaknesses
(-) The current metrics do not adequately capture persistent global consistency. Marginal FID and local diversity can remain favorable even when spatial inconsistencies exist. Among the reported metrics, LCC measures connectivity rather than the geometric correctness of revisited regions. Have the authors considered quantifying revisit consistency, loop-closure error, and road validity before and after fusion to provide more direct evidence for the global consistency claim?
(-) The evidence for dynamic simulation remains more limited than that for environment generation. Table 2 identifies the controller and initialization as major error sources, and Appendix 7.2 reports closed-loop collision statistics. However, downstream data collection limits each scene to six agents and retains only collision-free clips, while the learned policy is evaluated only in an open-loop setting. Could the authors consider adding closed-loop evaluations of the learned policy, particularly in denser traffic and more interactive scenarios?
Questions for authors
-
How consistent are overlapping or revisited regions before map fusion, and how much does the fusion procedure improve or alter them?
-
What fraction of generated simulation clips is discarded by collision filtering, and how does filtering change the behavior distribution?
-
Do downstream gains persist under matched optimizer steps and when training and evaluation are repeated across independently generated maps?
4 comments
Response to Reviewer Jjfj (2/2)
3. Sensitivity of downstream gains to optimizer steps and generated maps
Do downstream gains persist under matched optimizer steps and when training and evaluation are repeated across independently generated maps?
Thank you for raising this. We address the two factors separately.
Matched optimizer steps. To separate the effect of data from that of training length, we retrain OccWorld on the 1× OccSim data for 5× the optimizer steps, matching the 5× setting in Tab. 3:
| Training data | Optimizer steps | IoU (1 s / 2 s / 3 s) | mIoU (1 s / 2 s / 3 s) |
|---|---|---|---|
| OccSim 1× | N (Tab. 3) | 20.53 / 15.48 / 13.24 | 13.51 / 9.56 / 7.61 |
| OccSim 1× | 5N (new) | 21.08 / 15.31 / 13.41 | 14.01 / 9.60 / 7.54 |
| OccSim 5× | 5N (Tab. 3) | 23.87 / 16.92 / 13.47 | 19.54 / 12.31 / 9.08 |
Training longer on the same data brings no meaningful gain: all changes are within 0.6 points, and some horizons slightly decrease, indicating that the 1× model has already converged. Under matched optimizer steps, the 5× data still improves mIoU by 5.53 / 2.71 / 1.54 at 1 s / 2 s / 3 s and IoU by 2.79 / 1.61 at 1 s / 2 s, while IoU at 3 s is on par. The gains from scaling OccSim data therefore come from the additional data rather than from more optimization.
Independently generated maps. To verify that the downstream gains do not depend on a particular generated map, we independently generate two additional maps with different initial frames and random seeds, keeping the trajectory length and data collection protocol unchanged, and train OccWorld on each under the 1× setting:
| Training data | Metric | Map 1 (Tab. 3) | Map 2 (new) | Map 3 (new) | Mean ± std |
|---|---|---|---|---|---|
| OccSim 1× | IoU | 20.53 / 15.48 / 13.24 | 19.99 / 14.94 / 13.15 | 21.01 / 15.67 / 13.03 | 20.51 ± 0.51 / 15.36 ± 0.38 / 13.14 ± 0.11 |
| mIoU | 13.51 / 9.56 / 7.61 | 14.20 / 9.15 / 7.59 | 13.59 / 9.61 / 7.33 | 13.77 ± 0.38 / 9.44 ± 0.25 / 7.51 ± 0.16 | |
| CARLA (Uni-C), all maps | IoU | – | – | – | 18.31 / 14.09 / 12.03 |
| mIoU | – | – | – | 11.79 / 8.35 / 6.75 |
Entries are at 1 s / 2 s / 3 s; std is the sample standard deviation over the three maps.
Note that the CARLA data is already collected over all available CARLA maps, so the per-map columns are left blank for it, whereas each OccSim run uses a single generated map. The results are highly consistent across maps, and the map used in Tab. 3 lies close to the mean. Each of the three maps outperforms CARLA at every horizon on both IoU and mIoU, with a gap to CARLA 3–10× larger than the standard deviation across maps.
4. Simulation with higher agent density
How does the simulator behave in denser traffic, and can the learned policy be evaluated in closed loop?
Tab. 7 (Sec. 7.2) reports the effect of traffic density from 4 to 7 agents: the collision rate at 10 s rises from 0.12 to 0.41, and at 60 s from 0.49 to 0.79. This increase stems from the rule-based controller rather than the generated environment: each agent drives toward a randomly selected exit, and the 2D-IDM has no intersection protocol such as right-of-way or signal phases, so agents increasingly block each other as density grows. Dense, highly interactive traffic therefore calls for a stronger controller, which OccSim is designed to accommodate: the controller is a replaceable module, and since the environment is pre-built and the ego vehicle is part of the agent set (Alg. 6, l.6), a learned policy can replace IDM for the ego or for other agents without modifying environment generation. We agree that closed-loop evaluation of learned policies in dense traffic is an important next step, and have added this discussion to the limitations of IDM-based agent behavior simulation in Sec. 10.
We hope these clarifications, revisions and new experiments address your concerns. If so, we would be grateful if you would consider updating your assessment. We are happy to clarify any remaining questions during the discussion period!
Dear Reviewer Jjfj,
As the discussion period ends tomorrow, we would like to kindly follow up on our response. Following your suggestions, we have added new experiments for each of your questions:
- Revisit and loop-closure agreement (General Response): pairwise agreement of raw frames before fusion against a GT reference, showing that disagreement does not accumulate along the rollout, and the agreement at the loop-closure seam, which exceeds that of GT frame pairs with comparable overlap.
- Effect of fusion (point 1): fusion alters GT and generated keyframes to a comparable extent, and raises LCC from 74.1% to 92.5% for W-DiT.
- Collision filtering (point 2): the fraction of discarded clips and its effect on the behavior distribution, showing that filtering removes controller artifacts rather than selecting favorable behavior.
- Robustness of downstream gains (point 3): under matched optimizer steps and across three independently generated maps, the gains over CARLA persist with low variance.
For closed-loop evaluation of learned policies in dense traffic, we have clarified the scope of the current controller and added it to the limitations. We would greatly appreciate it if you could let us know whether these results address your concerns. If they do, we would be grateful if you would consider updating your assessment.
Thank you again for your time and constructive feedback.
Dear Reviewer Jjfj,
We have now incorporated all changes into the revised PDF, marked in blue. In particular, the agreement of overlapping and revisited regions before fusion is reported in Sec. 4.2 (Tab. R1), the effect of fusion in Sec. 7.10 (Tab. R5–R6), and the robustness of downstream gains in Sec. 7.9 (Tab. R3–R4); a summary of all revisions is given in our comment "Summary of revisions".
We would sincerely appreciate it if you could take a look at the revision and, if your concerns have been addressed, consider updating your assessment. Thank you again for your time and constructive feedback.
Response to Reviewer Jjfj (1/2)
We thank Reviewer Jjfj for the constructive review, and for recognizing the significance of the problem and the simulation pipeline we build. We have addressed the weaknesses raised regarding the fusion stage and the downstream simulation with new experiments and revisions in the paper. Since the weaknesses and questions overlap, we reorganize them into the following points and answer each in turn:
1. The effect of fusion on occupancy fidelity
We answer this in two parts.
Alteration. Applying the same fusion procedure to GT keyframes from the val set and to W-DiT rollouts, we compare each fused map with its raw keyframes:
These changes reflect fusing view-limited per-frame occupancy rather than correcting generation errors: fusion alters GT keyframes to a comparable extent, and removes even fewer voxels from W-DiT keyframes than from GT ones.
Improvement. Following the straight-trajectory setting in the paper, we roll out W-DiT for 200 frames and compare the map obtained by directly stitching the raw keyframes, which do not overlap, with the map after fusion:
In terms of 2D-FID, fusion does not improve fidelity, since the reference samples are per-frame labels before densification; the increase reflects the loss of sensor-like sparsity rather than degraded content, as it occurs for GT as well. Fusion does, however, improve LCC, since the same densification restores connectivity: sparse far-range observations leave holes that break directly stitched keyframes into disconnected fragments, and fusion fills them with votes from the non-keyframes.
2. Further analysis of clips discarded due to collisions
This is reported in Tab. 7 (Sec. 7.2). For downstream data collection we use at most 6 agents and 10 s clips, for which 38% of the clips contain a collision and are discarded. These collisions mainly stem from the rule-based IDM controller and layout initialization rather than the generated maps, consistent with Tab. 2. To measure how filtering changes the behavior distribution, we compare it with two alternatives: truncating each clip at its first collision, and keeping the full 10 s clip regardless of collisions.
JSD (×10⁻²) to real behavior, following the protocol of Tab. 2.
Keeping full clips introduces a simulation artifact: after a collision, the IDM followers stop behind the crashed vehicles, so traffic jams completely and the clip is dominated by near-zero speeds and abnormally small gaps. Truncating at collision avoids the jam but keeps the pre-collision behavior, and yields a distribution close to that of discarding. Discarding collided clips therefore removes an artifact of the rule-based controller rather than selecting favorable behavior.