- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to address the information bottleneck of trajectory-based traffic simulators, which discard the sensory geometry and occlusion context shaping real driving behavior. AutoWorld grounds multi-agent simulation in raw LiDAR observations via a self-supervised latent occupancy world model, uses the predicted occupancies to condition a coarse-to-fine diffusion motion model, and applies a cascaded quality-weighted DPP sampling strategy for diverse yet plausible rollouts at inference. On nuPlan and Waymo, AutoWorld improves generation realism and physical plausibility, remains reactive to planner-controlled ego behavior, and supports controllable, diverse safety-critical scenario generation.
Critical strengths
- The critique that trajectory abstractions are a lossy compression discarding the sensory cause of behavior is rarely tackled before.
- The paper reports gains across two benchmarks and five structured questions, and the occlusion study isolates the benefit of sensor grounding over trajectory-only and direct-LiDAR-conditioning baselines.
Critical weaknesses
- Single-modality (LiDAR-only) grounding limits generality and realism claims. The world model uses only LiDAR occupancy and leaves camera/semantic cues to future work. Yet driving behavior is strongly shaped by visual semantics (signs, lane markings, object classes), so it is unclear how much of the sensory context claim survives beyond geometric/occlusion cues, and the comparison against multimodal baselines is absent.
- The paper notes realism metrics differ per benchmark, and physical-plausibility metrics are aggregated differently across papers. This makes cross-method comparison fragile and raises questions about how robust the claimed SOTA status is to evaluation-protocol variation.
- AutoWorld couples a full latent occupancy world model with a conditional diffusion motion model and two-stage DPP. Neither training compute, inference latency, nor memory footprint is quantified, so it is unclear whether the realism gains justify the overhead versus cheaper trajectory-based baselines—especially for large-scale simulation.
- The closed-loop reactiveness test uses only a single-step planner interaction, leaving open whether the framework remains stable and reactive in longer, fully closed-loop multi-agent settings.
- Occlusion is estimated via BEV line-of-sight ray casting from the ego, and agents’ trajectories are taken as given from offline auto-labeling. This is a proxy for real sensor degradation and it is unclear whether AutoWorld’s advantage holds under realistic sensor noise or imperfect tracking, and whether the world model would be robust to such perturbations.
Questions for authors
- How does the framework behave under realistic sensor imperfections rather than geometric occlusion?
- Does the autoregressive world model remain accurate and the diffusion model stable for long-horizon simulation, and does the cascaded DPP sampling still scale computationally there
- What is the actual computational cost of the full pipeline, and does it justify the gains?
4 comments
Response to Reviewer p11Q - (1/2)
1. Computational Cost and Cost–Benefit Analysis
What is the computational cost of the full pipeline, and is it justified by the gains?
Thank you for this question, which led us to quantify AutoWorld’s computational cost and clarify its performance gains relative to computational overhead. We agree that this analysis is missing from the paper. AutoWorld takes 146 ms per rollout, compared with 99 ms for Nexus, while reducing the collision rate by ∼70% on nuPlan (27.19% → 8.14%, Table 1). DPP guidance accounts for only 7.3% of this runtime. Training requires 193 A100 GPU-hours.
We measure the end-to-end latency of closed-loop rollouts on nuPlan using a single NVIDIA RTX PRO 6000 Blackwell GPU. For every method, rollouts are generated in batches of 32, and we report the amortized time per rollout as mean ± std over 3 runs. Nexus is run with its official code, checkpoint and default sampling settings. Nexus takes 99.1 ± 0.6 ms per rollout. AutoWorld takes 146.3 ± 0.3 ms with cascaded DPP and 135.6 ± 0.9 ms with IID sampling, so DPP guidance adds 10.7 ms per rollout (7.3%). For this added cost, AutoWorld reduces the collision rate by 70% relative to Nexus on nuPlan (27.19% → 8.14%, Table 1). Training requires 193 A100 GPU-hours.
We will add a subsection to Appendix C (Computational Efficiency and Cost–Benefit Analysis) reporting the runtime comparison with Nexus and the breakdown of DPP's cost.
2. Closed-loop Reactiveness
The closed-loop reactiveness test uses only a single-step planner interaction, leaving open whether the framework remains stable and reactive in longer, fully closed-loop multi-agent settings.
Thank you for this question. We would like to clarify that the reactiveness test in Sec. 4.3 is actually fully closed-loop for multiple steps, not a single-step interaction. As stated in the experimental setup (Sec. 4, l. 302–304), simulation is receding-horizon over 8 s at 1 Hz.
We will restate this in Sec. 4.3 to avoid ambiguity.
3. DPP Scaling with Simulation Horizon
Does the cascaded DPP sampling scale computationally with the length of the simulation horizon?
Thank you for raising this question. Cascaded DPP sampling scales linearly with the simulation horizon, exactly as IID sampling does, and DPP's share of the runtime stays constant at 7.3%.
Simulation is receding-horizon: a longer horizon only adds replanning steps, and each step does the same work. DPP guidance adds no sampling steps; at each existing step, it adds one batched pass of the GRU trajectory encoder and the log-determinant gradients of one 4×4 and four 8×8 kernels. These sizes depend only on the number of world-model rollouts (N=4) and motion samples (M=8), not on the horizon. DPP’s cost per replanning step is therefore fixed, so its share of the runtime stays constant as the horizon grows: at our 8 s horizon, it adds 10.7 ms to the 146.3 ms per rollout (7.3%).
We will add a subsection to Appendix C (Computational Efficiency and Cost–Benefit Analysis) and clarify how DPP scales with the simulation horizon.
4. Handling Sensor Noise and Tracking Errors
Occlusion is estimated via BEV line-of-sight ray casting from the ego, and agents’ trajectories are taken as given from offline auto-labeling. This is a proxy for real sensor degradation and it is unclear whether AutoWorld’s advantage holds under realistic sensor noise or imperfect tracking.
Thank you for this question. The BEV line-of-sight ray casting is used only to assign occlusion levels for evaluation (Sec. 4.2); the world model itself is trained on visibility masks computed by 3D ray casting from real LiDAR returns (Appendix C.2). It therefore already learns from realistic sensor imperfections, such as sparse returns at range, dropouts and partially observed agents, rather than from idealized geometry. Tracking errors affect all methods, but AutoWorld's world model does not depend on tracks.
The world model does not depend on agent tracks. Agent trajectories from offline auto-labeling are the standard input of WOSAC and nuPlan, and every baseline uses the same tracks, so the comparison is consistent. Errors in these tracks affect all trajectory-based simulators, since tracks are their only view of the scene. AutoWorld is affected through the motion model's trajectory input, but the world model never uses agent tracks or identities; it forecasts occupancy from LiDAR alone. An object that the tracker misses or tracks poorly still produces LiDAR returns and appears in the occupancy grid. We therefore expect the world-model path to partly compensate for tracking errors, although we have not tested this with an onboard tracker.
Response to Reviewer p11Q - (2/2) (updated)
5. Differences in Realism Metrics Across Benchmarks
The paper notes realism metrics differ per benchmark, and physical-plausibility metrics are aggregated differently across papers. This makes cross-method comparison fragile and raises questions about how robust the claimed SOTA status is to evaluation-protocol variation.
Thank you for raising this point. To keep comparisons consistent, we follow each benchmark's established metric conventions, as in prior work, and do not compare realism scores across benchmarks, since their definitions differ (Sec.~4). The two benchmarks also measure realism in complementary ways. nuPlan uses the conventional distribution-based approach, measuring the distance between the distributions of driving statistics in simulated and logged rollouts. WOSAC, to our knowledge the only benchmark of its kind, instead uses a likelihood-based metric that scores how likely the logged behavior is under the distribution of simulated rollouts. Evaluating on both lets us assess AutoWorld's realism from both perspectives. If there is a specific protocol or aggregation the reviewer would like to see, we would be glad to report AutoWorld under it.
6. LiDAR-only grounding and visual semantics
Single-modality (LiDAR-only) grounding limits generality and realism claims. The world model uses only LiDAR occupancy and leaves camera/semantic cues to future work. Yet driving behavior is strongly shaped by visual semantics (signs, lane markings, object classes), so it is unclear how much of the sensory context claim survives beyond geometric/occlusion cues
We appreciate the opportunity to clarify this point. we agree that driving behavior is strongly shaped by semantics. AutoWorld’s benefit is not limited to geometry: when the world model carries semantics, AutoWorld exploits them, lowering the collision rate by 45–56% at every occlusion level, including low occlusion, where geometric and visibility cues matter least. Cameras provide this semantic information in raw sensor data, without the labels our experiment used, which makes a multimodal world model a natural next extension of AutoWorld.
a. AutoWorld benefits from semantics in the world model, not only from geometry.
To isolate the effect of feeding semantics into the world model, we trained it under three supervision schemes: semantic occupancy labels (Sem-RF; object classes such as vehicle, pedestrian, cyclist and construction cone), label-free occupancy with uniform supervision (NSem-RF), and our label-free motion-aware objective (NSem-MA-RF). The motion-model architecture and training are the same across variants, and DPP is disabled (IID sampling at both stages), so differences come only from what the world model encodes. We report the collision rate on the occlusion-balanced nuPlan scenarios of Sec. 4.2, averaged over 3 training seeds.
| World-model supervision | Collision (%), Low-Occlusion ↓ | Collision (%), Medium-Occlusion ↓ | Collision (%), High-Occlusion ↓ |
|---|---|---|---|
| Sem-RF (semantic labels, uniform) | 6.02 ± 0.41 | 9.72 ± 0.68 | 12.68 ± 0.96 |
| NSem-RF (label-free, uniform) | 13.62 ± 0.33 | 17.64 ± 1.09 | 26.13 ± 0.88 |
| NSem-MA-RF (label-free, motion-aware) | 10.16 ± 0.68 | 13.20 ± 0.81 | 15.35 ± 1.11 |
Adding semantic labels while keeping the same rectified-flow objective (NSem-RF Sem-RF) lowers the collision rate by 56%, 45% and 51% at low, medium and high occlusion. The large gain at low occlusion, where few agents are hidden, shows that AutoWorld uses the semantic content of the occupancy, not only its geometry and visibility. Our label-free motion-aware objective recovers 80% of this gain under high occlusion but only 46% under low occlusion. The remaining gap at low occlusion is what semantics add beyond motion and geometry.
b. Cameras are a label-free source of scene semantics, so multimodal grounding is a natural extension.
Sem-RF obtains semantics from annotated occupancy. Camera images carry semantic information such as object classes, signs and construction zones, as well as finer cues like turn signals and brake lights, in the raw sensor data without annotation. Because the motion model conditions on world-model occupancy rather than raw LiDAR, semantic occupancy predicted by a camera–LiDAR world model fits the same interface without changes to the motion model. The result above suggests that AutoWorld would benefit from it.
We will revise the paper to scope our claim to LiDAR grounding, and discuss grounding in other sensor modalities, such as cameras, as a future direction.
Follow-up with Reviewer p11Q - Invitation for Further Feedback
As the end of the discussion period approaches, we would like to follow up on our response. We have conducted additional experiments and provided clarifications that address the concerns raised by Reviewer p11Q. We would appreciate it if the reviewer could take a look at our responses and share any further feedback. We would be glad to answer any remaining questions before the discussion period ends.
We thank the reviewer again for the time and constructive feedback.
Response to Reviewer p11Q - Summary (updated)
We thank Reviewer p11Q for the review, and for recognizing the importance of addressing the information lost in trajectory-based abstractions, our consistent gains across benchmarks and evaluation dimensions, and the insights from our occlusion study. We have conducted additional experiments and provided clarifications that directly address the reviewer's concerns by (1) quantifying AutoWorld's compute cost against Nexus, (2) clarifying that the reactiveness test is fully closed-loop, (3) analyzing how DPP's cost scales with the simulation horizon, (4) explaining how AutoWorld handles sensor noise and tracking errors, (5) clarifying how the two benchmarks measure realism, and (6) evaluating the effect of adding semantics to the world model.
1. AutoWorld reduces the collision rate by ∼70% relative to Nexus, at 1.48× its runtime. On nuPlan, AutoWorld lowers the collision rate from 27.19% to 8.14%, and takes 146 ms per rollout vs. 99 ms for Nexus. DPP guidance accounts for only 7.3% of this runtime, and training requires 193 A100 GPU-hours.
2. The reactiveness test is fully closed-loop. It runs over multiple replanning steps (8 s at 1 Hz) in Sec. 4.3, not a single-step interaction.
3. DPP's share of runtime does not grow with the simulation horizon. Cascaded DPP sampling scales linearly with the simulation horizon, exactly as IID sampling does, and its share stays constant at 7.3%.
4. The world model learns from real sensor data and never uses agent tracks. It is trained on real LiDAR returns, with their sparsity, dropouts and partial views; the BEV occlusion proxy is used only to assign occlusion levels for evaluation. Tracking errors, which affect all trajectory-based methods, therefore do not reach AutoWorld's world model.
5. AutoWorld is evaluated under two complementary realism protocols. We follow each benchmark's conventions: nuPlan measures realism with a distribution-based metric and WOSAC with a likelihood-based one. We would be happy to evaluate AutoWorld under any alternative protocol the reviewer suggests.
6. AutoWorld benefits from semantics, not only geometry. Supervising its world model with semantic labels lowers the collision rate by 45–56% at every occlusion level, including low occlusion, where geometric and visibility cues matter least. Cameras carry such semantics in raw sensor data without labels, making multimodal world models a natural extension of AutoWorld.
Please find our detailed responses below.