acceptodds
Under review as a conference paper at ICLR 2027

AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Models

Abstract

Simulation with realistic traffic agents is essential for validating autonomous driving systems. Existing data-driven simulators learn agent behavior from higher-level abstractions such as 3D bounding boxes and polylines, inferred by upstream perception pipelines. These lossy abstractions discard sensory context that directly shapes agent behavior, limiting the distributional realism that simulation aims to reproduce. To address this limitation, we propose AutoWorld, a traffic simulation framework that grounds agent behavior in raw sensor observations through a self-supervised world model trained on LiDAR occupancy data. Given world model samples, AutoWorld constructs a coarse-to-fine predictive scene context as input to a multi-agent motion generation model. Furthermore, we designed a motion-aware latent supervision objective that enriches AutoWorld's latent representation of scene dynamics. To better exploit this latent space during inference, AutoWorld employs a cascaded Determinantal Point Process framework to guide diversity-aware sampling across both the world model and motion model. Experiments on nuPlan and Waymo demonstrate that AutoWorld improves generation realism, with larger gains in partially observed scenarios where trajectory abstractions are most limited, reducing collision rates by 36%. AutoWorld also remains physically plausible under planner-controlled ego behavior and supports controllable, diverse safety-critical scenario generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Traffic Simulation, World Models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_p11QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to address the information bottleneck of trajectory-based traffic simulators, which discard the sensory geometry and occlusion context shaping real driving behavior. AutoWorld grounds multi-agent simulation in raw LiDAR observations via a self-supervised latent occupancy world model, uses the predicted occupancies to condition a coarse-to-fine diffusion motion model, and applies a cascaded quality-weighted DPP sampling strategy for diverse yet plausible rollouts at inference. On nuPlan and Waymo, AutoWorld improves generation realism and physical plausibility, remains reactive to planner-controlled ego behavior, and supports controllable, diverse safety-critical scenario generation.

Critical strengths
  • The critique that trajectory abstractions are a lossy compression discarding the sensory cause of behavior is rarely tackled before.
  • The paper reports gains across two benchmarks and five structured questions, and the occlusion study isolates the benefit of sensor grounding over trajectory-only and direct-LiDAR-conditioning baselines.
Critical weaknesses
  • Single-modality (LiDAR-only) grounding limits generality and realism claims. The world model uses only LiDAR occupancy and leaves camera/semantic cues to future work. Yet driving behavior is strongly shaped by visual semantics (signs, lane markings, object classes), so it is unclear how much of the sensory context claim survives beyond geometric/occlusion cues, and the comparison against multimodal baselines is absent.
  • The paper notes realism metrics differ per benchmark, and physical-plausibility metrics are aggregated differently across papers. This makes cross-method comparison fragile and raises questions about how robust the claimed SOTA status is to evaluation-protocol variation.
  • AutoWorld couples a full latent occupancy world model with a conditional diffusion motion model and two-stage DPP. Neither training compute, inference latency, nor memory footprint is quantified, so it is unclear whether the realism gains justify the overhead versus cheaper trajectory-based baselines—especially for large-scale simulation.
  • The closed-loop reactiveness test uses only a single-step planner interaction, leaving open whether the framework remains stable and reactive in longer, fully closed-loop multi-agent settings.
  • Occlusion is estimated via BEV line-of-sight ray casting from the ego, and agents’ trajectories are taken as given from offline auto-labeling. This is a proxy for real sensor degradation and it is unclear whether AutoWorld’s advantage holds under realistic sensor noise or imperfect tracking, and whether the world model would be robust to such perturbations.
Questions for authors
  • How does the framework behave under realistic sensor imperfections rather than geometric occlusion?
  • Does the autoregressive world model remain accurate and the diffusion model stable for long-horizon simulation, and does the cascaded DPP sampling still scale computationally there
  • What is the actual computational cost of the full pipeline, and does it justify the gains?
4 comments
AuthorsOpenReview ↗

Response to Reviewer p11Q - Summary (updated)

We thank Reviewer p11Q for the review, and for recognizing the importance of addressing the information lost in trajectory-based abstractions, our consistent gains across benchmarks and evaluation dimensions, and the insights from our occlusion study. We have conducted additional experiments and provided clarifications that directly address the reviewer's concerns by (1) quantifying AutoWorld's compute cost against Nexus, (2) clarifying that the reactiveness test is fully closed-loop, (3) analyzing how DPP's cost scales with the simulation horizon, (4) explaining how AutoWorld handles sensor noise and tracking errors, (5) clarifying how the two benchmarks measure realism, and (6) evaluating the effect of adding semantics to the world model.

1. AutoWorld reduces the collision rate by ∼70% relative to Nexus, at 1.48× its runtime. On nuPlan, AutoWorld lowers the collision rate from 27.19% to 8.14%, and takes 146 ms per rollout vs. 99 ms for Nexus. DPP guidance accounts for only 7.3% of this runtime, and training requires 193 A100 GPU-hours.

2. The reactiveness test is fully closed-loop. It runs over multiple replanning steps (8 s at 1 Hz) in Sec. 4.3, not a single-step interaction.

3. DPP's share of runtime does not grow with the simulation horizon. Cascaded DPP sampling scales linearly with the simulation horizon, exactly as IID sampling does, and its share stays constant at 7.3%.

4. The world model learns from real sensor data and never uses agent tracks. It is trained on real LiDAR returns, with their sparsity, dropouts and partial views; the BEV occlusion proxy is used only to assign occlusion levels for evaluation. Tracking errors, which affect all trajectory-based methods, therefore do not reach AutoWorld's world model.

5. AutoWorld is evaluated under two complementary realism protocols. We follow each benchmark's conventions: nuPlan measures realism with a distribution-based metric and WOSAC with a likelihood-based one. We would be happy to evaluate AutoWorld under any alternative protocol the reviewer suggests.

6. AutoWorld benefits from semantics, not only geometry. Supervising its world model with semantic labels lowers the collision rate by 45–56% at every occlusion level, including low occlusion, where geometric and visibility cues matter least. Cameras carry such semantics in raw sensor data without labels, making multimodal world models a natural extension of AutoWorld.

Please find our detailed responses below.

AuthorsOpenReview ↗

Response to Reviewer p11Q - (1/2)

1. Computational Cost and Cost–Benefit Analysis

What is the computational cost of the full pipeline, and is it justified by the gains?

Thank you for this question, which led us to quantify AutoWorld’s computational cost and clarify its performance gains relative to computational overhead. We agree that this analysis is missing from the paper. AutoWorld takes 146 ms per rollout, compared with 99 ms for Nexus, while reducing the collision rate by ∼70% on nuPlan (27.19% → 8.14%, Table 1). DPP guidance accounts for only 7.3% of this runtime. Training requires 193 A100 GPU-hours.

We measure the end-to-end latency of closed-loop rollouts on nuPlan using a single NVIDIA RTX PRO 6000 Blackwell GPU. For every method, rollouts are generated in batches of 32, and we report the amortized time per rollout as mean ± std over 3 runs. Nexus is run with its official code, checkpoint and default sampling settings. Nexus takes 99.1 ± 0.6 ms per rollout. AutoWorld takes 146.3 ± 0.3 ms with cascaded DPP and 135.6 ± 0.9 ms with IID sampling, so DPP guidance adds 10.7 ms per rollout (7.3%). For this added cost, AutoWorld reduces the collision rate by 70% relative to Nexus on nuPlan (27.19% → 8.14%, Table 1). Training requires 193 A100 GPU-hours.

We will add a subsection to Appendix C (Computational Efficiency and Cost–Benefit Analysis) reporting the runtime comparison with Nexus and the breakdown of DPP's cost.

2. Closed-loop Reactiveness

The closed-loop reactiveness test uses only a single-step planner interaction, leaving open whether the framework remains stable and reactive in longer, fully closed-loop multi-agent settings.

Thank you for this question. We would like to clarify that the reactiveness test in Sec. 4.3 is actually fully closed-loop for multiple steps, not a single-step interaction. As stated in the experimental setup (Sec. 4, l. 302–304), simulation is receding-horizon over 8 s at 1 Hz.

We will restate this in Sec. 4.3 to avoid ambiguity.

3. DPP Scaling with Simulation Horizon

Does the cascaded DPP sampling scale computationally with the length of the simulation horizon?

Thank you for raising this question. Cascaded DPP sampling scales linearly with the simulation horizon, exactly as IID sampling does, and DPP's share of the runtime stays constant at 7.3%.

Simulation is receding-horizon: a longer horizon only adds replanning steps, and each step does the same work. DPP guidance adds no sampling steps; at each existing step, it adds one batched pass of the GRU trajectory encoder and the log-determinant gradients of one 4×4 and four 8×8 kernels. These sizes depend only on the number of world-model rollouts (N=4) and motion samples (M=8), not on the horizon. DPP’s cost per replanning step is therefore fixed, so its share of the runtime stays constant as the horizon grows: at our 8 s horizon, it adds 10.7 ms to the 146.3 ms per rollout (7.3%).

We will add a subsection to Appendix C (Computational Efficiency and Cost–Benefit Analysis) and clarify how DPP scales with the simulation horizon.

4. Handling Sensor Noise and Tracking Errors

Occlusion is estimated via BEV line-of-sight ray casting from the ego, and agents’ trajectories are taken as given from offline auto-labeling. This is a proxy for real sensor degradation and it is unclear whether AutoWorld’s advantage holds under realistic sensor noise or imperfect tracking.

Thank you for this question. The BEV line-of-sight ray casting is used only to assign occlusion levels for evaluation (Sec. 4.2); the world model itself is trained on visibility masks computed by 3D ray casting from real LiDAR returns (Appendix C.2). It therefore already learns from realistic sensor imperfections, such as sparse returns at range, dropouts and partially observed agents, rather than from idealized geometry. Tracking errors affect all methods, but AutoWorld's world model does not depend on tracks.

The world model does not depend on agent tracks. Agent trajectories from offline auto-labeling are the standard input of WOSAC and nuPlan, and every baseline uses the same tracks, so the comparison is consistent. Errors in these tracks affect all trajectory-based simulators, since tracks are their only view of the scene. AutoWorld is affected through the motion model's trajectory input, but the world model never uses agent tracks or identities; it forecasts occupancy from LiDAR alone. An object that the tracker misses or tracks poorly still produces LiDAR returns and appears in the occupancy grid. We therefore expect the world-model path to partly compensate for tracking errors, although we have not tested this with an onboard tracker.

AuthorsOpenReview ↗

Response to Reviewer p11Q - (2/2) (updated)

5. Differences in Realism Metrics Across Benchmarks

The paper notes realism metrics differ per benchmark, and physical-plausibility metrics are aggregated differently across papers. This makes cross-method comparison fragile and raises questions about how robust the claimed SOTA status is to evaluation-protocol variation.

Thank you for raising this point. To keep comparisons consistent, we follow each benchmark's established metric conventions, as in prior work, and do not compare realism scores across benchmarks, since their definitions differ (Sec.~4). The two benchmarks also measure realism in complementary ways. nuPlan uses the conventional distribution-based approach, measuring the distance between the distributions of driving statistics in simulated and logged rollouts. WOSAC, to our knowledge the only benchmark of its kind, instead uses a likelihood-based metric that scores how likely the logged behavior is under the distribution of simulated rollouts. Evaluating on both lets us assess AutoWorld's realism from both perspectives. If there is a specific protocol or aggregation the reviewer would like to see, we would be glad to report AutoWorld under it.

6. LiDAR-only grounding and visual semantics

Single-modality (LiDAR-only) grounding limits generality and realism claims. The world model uses only LiDAR occupancy and leaves camera/semantic cues to future work. Yet driving behavior is strongly shaped by visual semantics (signs, lane markings, object classes), so it is unclear how much of the sensory context claim survives beyond geometric/occlusion cues

We appreciate the opportunity to clarify this point. we agree that driving behavior is strongly shaped by semantics. AutoWorld’s benefit is not limited to geometry: when the world model carries semantics, AutoWorld exploits them, lowering the collision rate by 45–56% at every occlusion level, including low occlusion, where geometric and visibility cues matter least. Cameras provide this semantic information in raw sensor data, without the labels our experiment used, which makes a multimodal world model a natural next extension of AutoWorld.

a. AutoWorld benefits from semantics in the world model, not only from geometry.

To isolate the effect of feeding semantics into the world model, we trained it under three supervision schemes: semantic occupancy labels (Sem-RF; object classes such as vehicle, pedestrian, cyclist and construction cone), label-free occupancy with uniform supervision (NSem-RF), and our label-free motion-aware objective (NSem-MA-RF). The motion-model architecture and training are the same across variants, and DPP is disabled (IID sampling at both stages), so differences come only from what the world model encodes. We report the collision rate on the occlusion-balanced nuPlan scenarios of Sec. 4.2, averaged over 3 training seeds.

World-model supervisionCollision (%), Low-Occlusion ↓Collision (%), Medium-Occlusion ↓Collision (%), High-Occlusion ↓
Sem-RF (semantic labels, uniform)6.02 ± 0.419.72 ± 0.6812.68 ± 0.96
NSem-RF (label-free, uniform)13.62 ± 0.3317.64 ± 1.0926.13 ± 0.88
NSem-MA-RF (label-free, motion-aware)10.16 ± 0.6813.20 ± 0.8115.35 ± 1.11

Adding semantic labels while keeping the same rectified-flow objective (NSem-RF →\rightarrow Sem-RF) lowers the collision rate by 56%, 45% and 51% at low, medium and high occlusion. The large gain at low occlusion, where few agents are hidden, shows that AutoWorld uses the semantic content of the occupancy, not only its geometry and visibility. Our label-free motion-aware objective recovers 80% of this gain under high occlusion but only 46% under low occlusion. The remaining gap at low occlusion is what semantics add beyond motion and geometry.

b. Cameras are a label-free source of scene semantics, so multimodal grounding is a natural extension.

Sem-RF obtains semantics from annotated occupancy. Camera images carry semantic information such as object classes, signs and construction zones, as well as finer cues like turn signals and brake lights, in the raw sensor data without annotation. Because the motion model conditions on world-model occupancy rather than raw LiDAR, semantic occupancy predicted by a camera–LiDAR world model fits the same interface without changes to the motion model. The result above suggests that AutoWorld would benefit from it.

We will revise the paper to scope our claim to LiDAR grounding, and discuss grounding in other sensor modalities, such as cameras, as a future direction.

AuthorsOpenReview ↗

Follow-up with Reviewer p11Q - Invitation for Further Feedback

As the end of the discussion period approaches, we would like to follow up on our response. We have conducted additional experiments and provided clarifications that address the concerns raised by Reviewer p11Q. We would appreciate it if the reviewer could take a look at our responses and share any further feedback. We would be glad to answer any remaining questions before the discussion period ends.

We thank the reviewer again for the time and constructive feedback.

Reviewer_KFMEOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a self-supervised trajectory generation framework using LiDAR occupancy as latent states. The pipeline adopts motion-aware LiDAR embeddings followed by DPP-style diversity encouragement in the diffusion sampling process for both world-state and trajectory evolution. Extensive experiments over large-scale datasets demonstrate the performance of the proposed method.

Critical strengths
  1. The paper proposes incorporates LiDAR occupancy for latent world-state embeddings to guide the conditional diffusion process for trajectory generation, which is well-motivated.
  2. The effectiveness of the method is verified over simulation realism, occlusions, reactiveness, and diversity, which aligns with the key claims.
Critical weaknesses

The main unclarity is the interpretation of DPP as a "sampling process".

As described in Sec-3.4, the sampling mechanism of the world-state and the trajectory follows standard world-model and diffusion dynamics, where a diversity term is added based on the gradient of negative correlation that is used in DPP. It seems that no sampling is conducted directly over the distribution defined by DPP in mathematical sense.

In such case, there could be more clarity if the method framed as a "diversity encouraging" sampling method.

Questions for authors

Main Questions:

  1. If DPP is mainly adopted to encourage sampling diversity, could other diversity-promoting methods be applied instead of DPP? How is the diversity strength selected or tuned to avoid hurting realism?
  2. Sec-4.2 reports substantial improvements in scenarios with occlusions. When the trajectories are occluded, would the LiDAR view also be occluded at the same time? What extra information can be provided if so which contributes to improved model performances?

Other Clarifications:

  1. Could the authors indicate the compute requirements for training and inference?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KFME - Summary

We thank Reviewer KFME for the review, and for finding the use of LiDAR occupancy as a latent world state well motivated and our evaluation across realism, occlusion, reactiveness and diversity aligned with our claims. We have conducted additional experiments and provided clarifications that directly address all of the reviewer's concerns by (1) comparing DPP with a simpler diversity mechanism and reframing it as diversity-encouraging guidance, (2) clarifying the sources of AutoWorld's gains under occlusion, and (3) quantifying its training and inference compute.

1. DPP improves diversity over a simpler alternative at negligible extra cost. Compared with pairwise repulsive guidance, DPP improves diversity by 31–45% and achieves better realism. We agree that diversity-encouraging guidance is a clearer framing and will revise the paper accordingly.

2. AutoWorld's gains under occlusion come from observed geometry and learned forecasting, not from recovering hidden agents. Under high occlusion, these lower the collision rate from 58.01% (trajectory-only) to 42.68% (observed LiDAR) and 15.35% (AutoWorld).

3. Training requires 193 A100 GPU-hours, and inference takes 146 ms per rollout, of which DPP accounts for 7.3%.

Please find our detailed responses below.

AuthorsOpenReview ↗

Response to Reviewer KFME (1/2)

1. Framing of DPP Sampling

AutoWorld adds a DPP-based diversity gradient to standard world-model and diffusion sampling, rather than sampling from a DPP distribution. Would framing it as diversity-encouraging guidance be clearer? Could other diversity-promoting methods be applied instead of DPP? How is the diversity strength selected or tuned?

Thank you for this careful observation, which prompted us to revisit how we name our method and to provide additional implementation details. You are right that our method does not sample from the distribution a DPP defines, and diversity-encouraging guidance is a more accurate framing. AutoWorld is also compatible with other diversity-promoting mechanisms. Compared with pairwise repulsive guidance, one such alternative, DPP adds only the log-determinant of a small kernel and its closed-form gradient, while improving diversity by 31–45% and achieving better realism. The diversity strength follows a time-dependent schedule, with its peak tuned per dataset on held-out validation data.

a. Other diversity-promoting methods can replace DPP, but DPP adds little computation over repulsive guidance while improving diversity by 31–45%.

To compare cost and benefit, we extend the safety-critical scenario generation experiment (Sec. 4.4), which tests how well a method covers varied failure modes. We replace DPP with pairwise repulsive guidance using the same quality weights, guidance schedule, gradient normalization and rollout budget, applied at both stages (world model and motion model). Since repulsive guidance already uses the same encoder pass, pairwise kernels and quality weights, DPP additionally requires only a log-determinant and its closed-form gradient for each kernel. With our default 32 rollouts, these are one 4×4 kernel over the 4 world-model rollouts and four 8×8 kernels over the 8 motion samples of each rollout. Compared with repulsive guidance, DPP improves diversity by 30.9% in angle, 44.9% in relative speed, and 38.5% in collision point, while increasing target collisions by 6.8%, reducing off-road time by 39.1%, and improving realism (0.301 → 0.274). In contrast, repulsive guidance worsens realism relative to IID sampling (0.301 vs. 0.289). We attribute this to DPP concentrating guidance on redundant samples, with diminishing influence on distinct samples, whereas pairwise repulsion continues pushing even plausible, distinct samples apart.

b. The diversity strength is tuned once per dataset for realism. The diversity strength is a time-dependent coefficient, γμ(kμ)\gamma_\mu(k_\mu) for the world model and γf(kf)\gamma_f(k_f) for the motion model, following a power-cosine schedule (Gao et al., 2023). The schedule applies weak guidance early in sampling, when samples are still mostly noise, and stronger guidance toward the end. The DPP gradient is normalized by its norm, so the coefficient sets a bounded step size, and the gradient becomes negligible once samples are already diverse, so guidance acts only on redundant sets. The peak values were selected separately for each dataset on a small held-out validation set, choosing the value with the best realism score, and kept fixed for all experiments on that dataset.

ModelTarget Collision ↑\uparrowNon-Target Collision ↓\downarrowOff-road Time ↓\downarrowRealism ↓\downarrowAngle Var (rad\mathrm{rad}) ↑\uparrowRelative Speed Var (m/s\mathrm{m/s}) ↑\uparrowPoint Var (m\mathrm{m}) ↑\uparrow
AutoWorld−DPP34.569.425.280.2891.931.271.53
AutoWorld+DPP41.847.033.600.2743.774.586.19
AutoWorld+Repulsive Guidance39.177.085.910.3012.883.164.47

We will revise all five sections, particularly the method (Sec. 3.4), to frame DPP as diversity-encouraging guidance. We will add the clarifications on diversity strength to Appendix C.4 (Cascaded Latent Diversity), and the repulsive-guidance row and its setup to the table in Sec. 4.4.

AuthorsOpenReview ↗

Response to Reviewer KFME (2/2)

2. Clarifying LiDAR’s Benefits Under Occlusion

When the trajectories are occluded, would the LiDAR view also be occluded at the same time? What extra information can be provided if so which contributes to improved model performances?

Thank you for this question, which gives us an opportunity to further elaborate on the motivation for incorporating an occupancy-based world model into our simulation framework. An agent fully occluded from the ego-vehicle sensors is missing from the LiDAR view, so AutoWorld does not recover occluded agents directly. The benefit comes from two other sources of information that trajectory abstractions discard.

a. The smaller effect:

LiDAR captures occluding objects. Trajectory representations contain only tracked agents and map elements, so untracked structures are absent. LiDAR occupancy records this geometry, and with it where visibility is limited and where free space ends. This structure shapes behavior, for example slowing near blind corners or yielding near occluded crosswalks, and it determines the line of sight of every agent in the scene, not only the ego’s.

b. The larger effect:

Because the world model is trained on sequences in which regions hidden at one time are revealed later as the ego moves, the world model can learn where objects are likely to emerge from occluded areas, information that no single observation contains. To quantify these effects, we run a new experiment that compares three variants, all with DPP disabled: a trajectory-only model, a model conditioned directly on observed LiDAR, and AutoWorld conditioned on world-model forecasts. Under high occlusion, conditioning the motion model directly on LiDAR reduces the collision rate from 58.01% to 42.68%, reflecting the observed geometry. AutoWorld, which conditions on world-model forecasts, reduces it further to 15.35%, reflecting the learned forecasting.

VariantLiDAR ConditioningCollision rate (%) Low-Occlusion ↓\downarrowCollision (%) Medium-Occlusion ↓\downarrowCollision (%) High-Occlusion ↓\downarrow
Traj.-DiffusionNone17.32 ± 0.3929.41 ± 0.9558.01 ± 1.72
LiDAR-DiffusionPast (observed)16.86 ± 1.0623.43 ± 1.6842.68 ± 1.33
AutoWorld (IID)Future (forecast)10.16 ± 0.6813.20 ± 0.8115.35 ± 1.11

We will add this discussion to Sec. 4.2, stating explicitly that AutoWorld does not recover fully occluded agents and explaining the two sources of its gains under occlusion. We also report the results of this experiment in Appendix B.

3. Quantifying Training and Inference Costs

Could the authors indicate the compute requirements for training and inference?

Thank you for highlighting the need to clarify the compute requirement. Training requires 193 A100 GPU-hours. We measure the end-to-end latency of closed-loop rollouts on nuPlan using a single NVIDIA RTX PRO 6000 Blackwell GPU. Rollouts are generated in batches of 32, and we report the amortized time per rollout as mean ± std over 3 runs. AutoWorld takes 146.3 ± 0.3 ms per rollout with cascaded DPP and 135.6 ± 0.9 ms with IID sampling (DPP adds 10.7 ms, 7.3% of runtime)

We will report the training and inference compute requirements in Appendix C (Computational Efficiency and Cost–Benefit Analysis).

I would like to thank the authors for their response, which addresses my previous concerns. The paper remains consistent with my previous assessment.

Reviewer_zYyJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AutoWorld, a traffic-simulation framework that grounds multi-agent behavior generation in a self-supervised LiDAR world model rather than exclusively in trajectory abstractions (bounding boxes and polylines). The central argument is that trajectory-only supervision discards sensory context (geometry, occlusion) that shapes real driving behavior, and that a sensor- grounded world model can supply this context to the behavior generator.

AutoWorld has three components:

  1. a motion-aware latent occupancy forecasting world model trained on unlabeled LiDAR, with a motion-weighted rectified-flow objective (§3.2) that upweights dynamic voxels without requiring semantic labels;
  2. a world-model-grounded motion-diffusion decoder (§3.3) that is conditioned on both local per-step and global multi-step pooled features from the world model; and
  3. a cascaded diversity-guided sampling procedure (§3.4) based on a quality-weighted flow-space Determinantal Point Process , applied at both the world-model and the motion-model stages.

Experiments on nuPlan and the Waymo Open Sim Agents benchmark show state-of-the-art realism among imitation-trained methods, with a particularly large collision- rate reduction and favorable scaling with additional LiDAR data compared to a parameter-matched trajectory-diffusion baseline.

Critical strengths
  • Thoughtful self-supervision design. The motion-aware weight map (§3.2, Eq. after l.185) is an elegant mechanism to bias training toward dynamic voxels without semantic labels. Table 4 shows measurable dynamic IoU gains and consistent downstream RMM gains , approaching the label-based semantic baseline without the annotation cost.
  • Strong empirical results. On nuPlan (Table 1), AutoWorld improves Realism from the best prior diffusion method and reduces collision from the second-best. On Waymo (Table 2), AutoWorld matches or exceeds all imitation-trained baselines on RMM and matches RL-fine-tuned simulators without reward-based post-training.
  • The occlusion study is convincing result. Figure 4 shows the baseline-AutoWorld gap grows with occlusion intensity, consistent with the central hypothesis that sensor grounding helps most where trajectory context is weakest.
  • Comprehensive ablations. Table 5 decomposes world-model grounding, motion-level DPP, and occupancy-level DPP. Removing world-model grounding has the largest effect which supports the paper's central narrative.
Critical weaknesses
  • Contribution attribution is not sharp. Three nominally independent components are introduced (world model, grounding mechanism, DPP sampling). Table 5 attributes the largest single gain to the world model, but motion-aware supervision adds only a little on top of non-semantic supervision (Table 4), and the DPP mechanisms contribute smaller improvements. A reader could conclude that the primary signal is "add LiDAR features"; a clearer decomposition would help.
  • DPP machinery is heavy relative to its gain. The quality-weighted L-ensemble DPP, applied via gradient guidance into the flow ODE at two stages (§3.4), is a substantial addition to the pipeline. On the Waymo RMM headline (Table 2), the gain over CAT-K+ is modest . A fair comparison would be against a simpler diversity mechanism — e.g., temperature or nucleus sampling at the trajectory level, or a repulsive guidance scheme without DPP — to show DPP is doing something simpler mechanisms cannot.
  • No ablation of sensor modality. The paper argues sensor grounding helps, but only tests LiDAR occupancy. Does adding camera features help further? A multi-modal variant is listed as future work; even a small-scale pilot would substantially strengthen the generality claim.
  • Closed-loop reactiveness is relative, not absolute. Table 3 reports lower collision rates for AutoWorld under externally controlled ego planners, but AutoWorld still has non-trivial collision rates. An analysis of which planner-simulator interactions dominate the residual collisions would clarify what is left to improve.
  • Safety-critical generation evaluation is narrow. §4.4 compares only against Nexus under a single Safe-Sim-style configuration. A Pareto plot showing controllability vs. realism across guidance strengths would substantiate the "diverse and controllable" claim.
Questions for authors
  • For the occlusion experiment (Fig. 4), what is AutoWorld's collision rate if the world model is replaced with a semantic occupancy predictor trained with labels? This would isolate the benefit of LiDAR grounding from self-supervised LiDAR grounding.
  • How many flow ODE steps does DPP guidance add at inference? What is the end-to-end throughput compared to Nexus or SMART baselines?
  • In the controllable safety-critical experiment, AutoWorld+DPP improves Target Collision but Realism Score worsens . Does controllability trade off against realism in general? A Pareto plot across guidance strengths would clarify this.
  • nuScenes LiDAR has very different sweep patterns from Waymo / nuPlan. Portability across sensor rigs would strengthen the generality claim.

Minor issues

  • Equation 1 (p.4) and l.164 both use τ for different timesteps (flow-matching vs. state-sequence). Please disambiguate.
  • Table 4 shows Sem-RF RMM 0.7651 vs. NSem-MA-RF 0.7618. Without reported variance, "closes much of the gap" is hard to interpret. Please add seed variance.
  • Related work could note why GAIA, Vista, and DriveDreamer are not direct baselines (camera-based, generative) in one sentence rather than leaving the comparison implicit.
  • The reproducibility statement promises code upon acceptance. For a work whose central claim rests on the world model, releasing at least a world-model checkpoint would meaningfully strengthen the submission.
5 comments
AuthorsOpenReview ↗

Response to Reviewer zYyJ - Summary

We thank Reviewer zYyJ for the careful and detailed review, and for recognizing our self-supervised design, our empirical results, and how the occlusion analysis and ablations support our central hypothesis. We have conducted additional experiments and provided clarifications that directly address all of the reviewer's concerns by (1) disentangling the contributions of our framework, (2) analyzing computational efficiency and cost–benefit trade-offs, (3) evaluating the controllability–realism trade-off, (4) assessing generalization across sensor setups, (5) explaining why camera-based world models are not direct baselines, and (6) showing how AutoWorld can benefit from richer sensory information such as cameras.

1. AutoWorld's gains stem from forecasting rather than merely incorporating LiDAR features (15.35% vs. 42.68% collision rate under high occlusion). Moreover, our motion-aware objective reduces collision rates by 25.2–41.3% compared with the standard rectified flow objective and recovers up to 80% of the gains from semantic supervision without requiring semantic annotations.

2. DPP guidance adds no sampling steps and accounts for 7.3% of AutoWorld's end-to-end runtime. The full pipeline takes 146 ms per rollout, compared with 99 ms for Nexus (1.48×), while reducing the collision rate by 70% on nuPlan. Compared with pairwise repulsive guidance, DPP adds only the log-determinant of a small kernel and its closed-form gradient, while improving diversity by 31–45% and achieving better realism.

3. Increasing guidance strength introduces a trade-off between controllability and realism. However, AutoWorld offers a better controllability–realism trade-off than Nexus. At comparable target-collision rates, AutoWorld achieves a lower realism distance and fewer non-target collisions.

4. AutoWorld transfers across LiDAR rigs. With its world model trained on a different LiDAR dataset (Waymo) and kept frozen, it still reduces the collision rate on nuPlan when used as the backbone for trajectory diffusion, retaining ∼84% of the gain from a world model trained on nuPlan itself.

5. Camera-based world models are not direct baselines for AutoWorld. Models such as GAIA-1, Vista and DriveDreamer generate future videos, not the per-agent trajectories that closed-loop multi-agent simulation requires; we will state this explicitly in Sec. 2.

6. AutoWorld improves with more informative world models. In our decomposition experiment (item 1), supervising the world model with semantic labels lowers the collision rate under high occlusion from 26.13% to 12.68%. Cameras provide such semantics without labeling cost, which makes multimodal world models a natural extension of AutoWorld.

Please find our detailed responses below.

AuthorsOpenReview ↗

Response to Reviewer zYyJ (1/4)

1. Disentangling Contribution Attribution

The contributions of the world model, grounding, and DPP sampling aren’t clearly separated: results may suggest most of the gain comes from adding LiDAR features. A clearer breakdown of each component’s contribution is required. For example, for the occlusion experiment (Fig. 4), what is AutoWorld's collision rate if the world model is replaced with a semantic occupancy predictor trained with labels?

Thank you for raising this question, which led us to run a decomposition experiment the original paper was missing, and it sharpens our central claim. We agree that Table 5 cannot disentangle the effect of LiDAR conditioning from that of forecasting, and that the contribution of our motion-aware objective is confounded with LiDAR features. Through additional experiments, we demonstrate that AutoWorld's gains stem from forecasting rather than merely incorporating LiDAR features (15.35% vs. 42.68% collision rate under high occlusion). Moreover, our motion-aware objective reduces collision rates by 25.2–41.3% compared with the standard rectified flow objective and recovers up to 80% of the gains from semantic supervision without requiring semantic annotations.

We ran the suggested experiment with 3 training seeds per variant. All world-model variants use IID sampling, so DPP does not contribute to these numbers.

VariantLiDAR ConditioningWM SupervisionCollision rate (%) Low-Occlusion ↓\downarrowCollision rate (%) Medium-Occlusion ↓\downarrowCollision rate (%) High-Occlusion ↓\downarrow
Traj.-DiffusionNone--17.32 ± 0.3929.41 ± 0.9558.01 ± 1.72
LiDAR-DiffusionPast (observed)--16.86 ± 1.0623.43 ± 1.6842.68 ± 1.33
Sem-RFFuture (forecast)Labeled, uniform6.02 ± 0.419.72 ± 0.6812.68 ± 0.96
NSem-RFFuture (forecast)Label-free, uniform13.62 ± 0.3317.64 ± 1.0926.13 ± 0.88
NSem-MA-RFFuture (forecast)Label-free, motion-aware10.16 ± 0.6813.20 ± 0.8115.35 ± 1.11

a. Forecasting, not LiDAR features, drives the gain.

Under high occlusion, NSem-MA-RF reaches 15.35% collision, compared with 42.68% for LiDAR-Diffusion, which conditions directly on observed LiDAR. This holds for all three world-model variants, including label-free uniform supervision (12.7–26.1% vs. 42.7–58.0% for the two baselines without a world model). LiDAR features alone therefore do not explain the improvement. The gain comes from forecasting scene occupancy.

b. Motion-aware supervision contributes to reducing collisions.

Compared with the standard rectified flow objective (NSem-RF), our motion-aware objective (NSem-MA-RF) reduces collision rate by 25.2–41.3%, and the gain grows with occlusion: 3.5, 4.4 and 10.8 points at low, medium and high occlusion (26.13% → 15.35% under high occlusion). The gains reported in Table 4 are averaged across all scenes, which obscures this effect; our decomposition further clarifies the contribution.

c. Motion-aware supervision recovers most of the benefit of labels.

As you correctly pointed out, labels help: Sem-RF achieves lower collision than NSem-MA-RF at all occlusion levels. However, of the reduction that labels give over label-free training (NSem-RF →\rightarrow Sem-RF), motion-aware supervision recovers ∼46% (low), ∼56% (medium) and ∼80% (high) without any semantic annotations. AutoWorld places no constraint on how the world model is supervised, so a better occupancy forecaster, with or without labels, directly gives a better simulator. This further supports world-model grounding as the main contribution.

We will update Fig. 4 to include AutoWorld with IID sampling, so that the comparison with LiDAR-Diffusion isolates the effect of world-model forecasting from that of LiDAR features. We will also add the table above to Sec. 4.5 as an ablation of world-model supervision (Sem-RF, NSem-RF, NSem-MA-RF across occlusion levels).

AuthorsOpenReview ↗

Response to Reviewer zYyJ (2/4)

2. Computational Efficiency and Cost–Benefit Analysis

How many flow ODE steps DPP adds and how AutoWorld’s end-to-end throughput compares with Nexus? The paper didn’t compare with other simpler diversity methods (e.g., temperature or nucleus sampling at the trajectory level, or a repulsive guidance scheme without DPP).

Thank you for this question, which prompted us to quantify AutoWorld’s computational cost and clarify its performance gains relative to computational overhead. We agree that this analysis is missing from the paper. DPP guidance adds no sampling steps and accounts for 7.3% of AutoWorld's end-to-end runtime. The full pipeline takes 146 ms per rollout, compared with 99 ms for Nexus (1.48×), while reducing the collision rate by 70% on nuPlan. Compared with pairwise repulsive guidance, DPP adds only the log-determinant of a small kernel and its closed-form gradient, while improving diversity by 31–45% and achieving better realism.

a. DPP guidance adds no sampling steps

With or without DPP, the world model and motion model use the same flow steps and denoising steps; the DPP gradient is added to the update at each existing step. For the world model, we compute the gradient on the one-step target estimate with the predicted velocity held constant, so no gradient passes through the world model. For the motion model, the gradient is taken with respect to the predicted clean trajectory and shifts the reverse-step mean, as in guided diffusion (Zhong et al., 2023b), so no gradient passes through the denoiser. The only learned module involved is the GRU trajectory encoder, which embeds trajectories for the DPP kernel and requires one batched forward and backward pass per step. With our default 32 rollouts (N=4 world-model rollouts × M=8 motion samples), the remaining cost is one 4×4 kernel at the occupancy level and four 8×8 kernels at the motion level, whose log-determinant gradients are computed in closed form; quality weights are treated as constants.

b. AutoWorld adds 47 ms per rollout over Nexus (146 vs. 99 ms) while reducing the collision rate by 70%, and DPP accounts for only 7.3% of its runtime.

We measure the end-to-end latency of closed-loop rollouts on nuPlan using a single NVIDIA RTX PRO 6000 Blackwell GPU. For every method, rollouts are generated in batches of 32, and we report the amortized time per rollout as mean ±\pm std over 3 runs. Nexus is run with its official code, checkpoint and default sampling settings. Nexus takes 99.1 ± 0.6 ms per rollout. AutoWorld takes 146.3 ± 0.3 ms with cascaded DPP and 135.6 ± 0.9 ms with IID sampling, so DPP guidance adds 10.7 ms per rollout (7.3%). For this added cost, AutoWorld reduces the collision rate by 70% relative to Nexus on nuPlan (27.19% → 8.14%, Table 1).

c. DPP introduces limited additional computation over repulsive guidance while improving diversity by 31–45%

We extend the safety-critical scenario generation experiment (Sec. 4.4), which measures diversity by assessing how well a method covers different failure modes. We replace DPP with pairwise repulsive guidance using the same quality weights, guidance schedule, gradient normalization and rollout budget, applied at both stages (world model and motion model). Since repulsive guidance already uses the same encoder pass, pairwise kernels, and quality weights, DPP additionally requires only the log-determinant of each (4×4) or (8×8) kernel and its closed-form gradient. Compared with repulsive guidance, DPP improves diversity by 30.9% in angle, 44.9% in relative speed, and 38.5% in collision point, while increasing target collisions by 6.8%, reducing off-road time by 39.1%, and improving realism (0.301 → 0.274). In contrast, repulsive guidance worsens realism relative to IID sampling (0.301 vs. 0.289). We attribute this to DPP concentrating guidance on redundant samples, with diminishing influence on distinct samples, whereas pairwise repulsion continues pushing even plausible, distinct samples apart.

ModelTarget Collision ↑\uparrowNon-Target Collision ↓\downarrowOff-road Time ↓\downarrowRealism ↓\downarrowAngle Var (rad\mathrm{rad}) ↑\uparrowRelative Speed Var (m/s\mathrm{m/s}) ↑\uparrowPoint Var (m\mathrm{m}) ↑\uparrow
AutoWorld−DPP34.569.425.280.2891.931.271.53
AutoWorld+DPP41.847.033.600.2743.774.586.19
AutoWorld+Repulsive Guidance39.177.085.910.3012.883.164.47

We will add a subsection to Appendix C (Computational Efficiency and Cost–Benefit Analysis) reporting the runtime comparison with Nexus and the breakdown of DPP's cost. In Sec. 3.4 (DPP-Guided Sampling), we will state explicitly that DPP guidance adds no sampling steps. We will also add the repulsive-guidance row and its setup to the table in Sec. 4.4.

Reviewer_vyfWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AutoWorld, a framework that integrates a self-supervised occupancy world model into multi-agent traffic simulation. Specifically, it predicts future latent occupancies from LiDAR observations and uses them as conditioning signals for a diffusion-based multi-agent motion generation model. The paper further introduces motion-aware supervision and cascaded latent sampling to improve scene modeling and behavior generation. Experiments on nuPlan and Waymo demonstrate improvements in simulation realism.

Critical strengths
  1. The paper evaluates the proposed method on two widely used autonomous driving benchmarks and considers various aspects of traffic simulation, including realism, physical plausibility, reactiveness, and controllability. The ablation studies demonstrate the contributions of the proposed components and show that the introduced designs bring consistent performance improvements.
  2. Incorporating sensor-grounded scene representations into multi-agent traffic simulation is a meaningful direction, as trajectory-based representations may discard useful environmental information.
Critical weaknesses

None. The proposed framework largely builds upon existing techniques, including latent occupancy prediction and diffusion-based motion generation. The overall contribution somewhat incremental but I think the main idea is reasonable.

Questions for authors
  1. Since the method relies on ego-centric LiDAR observations, some agents may be partially or completely occluded. How are these unobserved regions handled when constructing occupancy grids? Are they treated as empty, unknown, or inferred using additional information?
  2. In a simulator, complete scene states are usually accessible, making it possible to construct ground-truth occupancy grids. Could the authors use available offline annotations to construct a more complete occupancy representation and compare it with the current LiDAR-based occupancy? This would help evaluate the impact of incomplete observations on the proposed method.
  3. Since the world model generates future occupancies autoregressively, prediction errors may accumulate over time. How does the model maintain stable predictions during long-horizon rollouts? Have the authors evaluated its performance over different rollout horizons?
4 comments
AuthorsOpenReview ↗

Response to Reviewer vyfW - Summary

We thank Reviewer vyfW for the review, and for recognizing the importance of grounding traffic simulation in sensor observations, our comprehensive evaluation, and the consistent gains in our ablations. We have conducted additional experiments and provided clarifications that directly address all of the reviewer's concerns by (1) specifying how unobserved regions are represented, (2) isolating the benefit of learning from partial occupancy, which explains why AutoWorld uses it by design, and (3) evaluating AutoWorld over longer rollout horizons.

1. Unobserved regions are treated as unknown, not as empty. A visibility mask computed by ray casting from raw LiDAR alone marks each voxel as occupied, observed free or unobserved; no labels or maps are used to fill unobserved voxels.

2. Partial occupancy is a deliberate design choice. It lets the world model learn where hidden agents are likely to emerge. A new experiment isolates this effect: under high occlusion, conditioning on world-model forecasts lowers the collision rate from 42.68% (observed LiDAR) to 15.35%.

3. When the horizon is doubled, AutoWorld degrades less than the trajectory-only baseline: its collision rate rises by 3.34 points, compared with 17.7 points.

Please find our detailed responses below.

AuthorsOpenReview ↗

Response to Reviewer vyfW (1/2)

1. Clarifying Occupancy Representation Under Occlusion

Since the method relies on ego-centric LiDAR observations, some agents may be partially or completely occluded. How are these unobserved regions handled when constructing occupancy grids? Are they treated as empty, unknown, or inferred using additional information?

Thank you for this question, which prompted us to provide additional implementation details. Unobserved regions are treated as unknown, not as empty. Each occupancy grid is paired with a LiDAR visibility mask computed by ray casting from the sensor origin to each LiDAR return (Appendix C.2). A voxel is occupied if it contains a LiDAR return, observed free if a ray passes through it, and unobserved if no ray reaches it, for example behind an occluding vehicle. No additional information, such as labels or maps, is used to fill unobserved voxels.

2. Clarifying the Impact of Incomplete Occupancy Observations

In a simulator, complete scene states are usually accessible, making it possible to construct ground-truth occupancy grids. Could the authors use available offline annotations to construct a more complete occupancy representation and compare it with the current LiDAR-based occupancy? This would help evaluate the impact of incomplete observations on the proposed method.

Thanks for raising this question. AutoWorld is built on partial occupancy by design, not as an approximation of complete occupancy. Partial occupancy is what lets the world model learn where hidden agents are likely to emerge, which accounts for most of AutoWorld's gain under occlusion (42.68% → 15.35% collision rate under high occlusion). It also matches what the logged drivers could see and what the model receives at test time.

a. Unobserved regions are what the world model learns from.

Each occupancy grid comes with a visibility mask, so the world model sees what is occupied, where observation ends, and which structure blocks it. In the training sequences, regions hidden at one time are revealed later as the ego moves; sometimes an agent emerges and sometimes nothing does. By pairing these outcomes with the occluder geometry, the world model learns a distribution over the hidden scene, not one filled-in state. With complete occupancy, every hidden agent would be visible in advance, so the model would never need to learn to anticipate occluded agents. To isolate the benefit of this learned forecasting, we run a new experiment comparing three variants, all with DPP disabled: a trajectory-only model, a model conditioned directly on observed LiDAR, and AutoWorld conditioned on world-model forecasts. Under high occlusion, conditioning on forecasts reduces the collision rate from 42.68% (observed LiDAR) to 15.35%, although the world model is trained only on partial occupancy.

VariantLiDAR ConditioningCollision rate (%) Low-Occlusion ↓\downarrowCollision (%) Medium-Occlusion ↓\downarrowCollision (%) High-Occlusion ↓\downarrow
Traj.-DiffusionNone17.32 ± 0.3929.41 ± 0.9558.01 ± 1.72
LiDAR-DiffusionPast (observed)16.86 ± 1.0623.43 ± 1.6842.68 ± 1.33
AutoWorld (IID)Future (forecast)10.16 ± 0.6813.20 ± 0.8115.35 ± 1.11

b. Real driving behavior responds to partial observation.

The drivers in the logs could not see behind occluders either. Slowing near a blind corner or yielding near an occluded crosswalk is caused by the occlusion itself, not by what happens to be behind it. Filling in hidden regions would show this cautious behavior without its cause. Since AutoWorld aims to reproduce real-world driving behavior, partial occupancy is the representation the motion model should condition on.

c. Complete occupancy is not available at test time.

At test time the world model receives only ego-centric LiDAR, so training on the same representation keeps training and test inputs consistent.

We will motivate partial occupancy as a deliberate design choice in Sec. 3, and add to Sec. 4.2 that it is the source of AutoWorld's advantage over trajectory-based baselines under occlusion. We also report the results of this experiment in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer vyfW (2/2)

3. Long-horizon Simulation Stability

How does AutoWorld maintain stable predictions during long-horizon rollouts? Have the authors evaluated its performance over different rollout horizons?

Thank you for this question. AutoWorld degrades less than the trajectory-only baseline when the rollout horizon is doubled: from H to 2H, its collision rate rises by 3.34 points and its realism by 0.028, compared with 17.7 points and 0.216 for the baseline.

We evaluate closed-loop simulation at H and 2H (H = 8 s) on the occlusion-balanced nuPlan scenarios of Sec. 4.2, with the world model rolled out autoregressively. Simulation beyond benchmark horizons is an open challenge for learned simulators. Since AutoWorld improves directly with its world model, advances in long-horizon world modeling will further improve its stability. We will add this analysis to Appendix B.

VariantHorizonRealism ↓\downarrowCollision rate (%) ↓\downarrow
Traj.-DiffusionH0.44334.91
Traj.-Diffusion2H0.65952.56
AutoWorld (Full)H0.27811.03
AutoWorld (Full)2H0.30614.37

Thank you for the authors' clarification. I believe their response has largely addressed my concerns. I will maintain my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Rebuttal: Reviewer Feedback, Additional Results, and Discussion Status (1/2)

Dear Area Chair and Reviewers,

We thank all four reviewers for their careful reviews and constructive feedback. Reviewers recognized the importance of addressing the information lost in trajectory abstractions (vyfW, p11Q) and found grounding simulation in LiDAR observations a well-motivated and meaningful direction (KFME, vyfW). They also found our self-supervised motion-aware design thoughtful (zYyJ), and our evaluation of realism, occlusion, reactiveness and controllability on two widely used autonomous driving benchmarks comprehensive (vyfW, zYyJ, KFME, p11Q).

In this work, we address the information lost in trajectory abstractions, which discard sensory context, such as scene geometry and occlusion, that shapes real driving behavior. (1) We propose AutoWorld, a framework that grounds traffic simulation in raw LiDAR observations through a self-supervised world model. (2) We introduce a motion-aware latent supervision objective that emphasizes dynamic scene elements, enabling self-supervised world-model training without semantic labels. (3) We propose a cascaded diversity-encouraging sampling, a training free inference-time approach that enhances the diversity of simulated scenarios while maintaining high realism. (4) Extensive experiments on nuPlan and Waymo demonstrate AutoWorld's improved generation realism, with larger gains under partial observability. AutoWorld is reactive to planner-induced ego behavior and enables controllable, diverse safety-critical scenario generation, while ablations validate its key components.

During the response period, we believe we addressed all the key concerns raised by reviewers and further improved our work. Reviewers vyfW and KFME confirmed that our responses addressed their concerns and maintained their original weak-accept scores. Reviewer zYyJ raised their score from weak reject to weak accept. We summarize the new experiments and clarifications below.

1. AutoWorld's gain comes from forecasting, not from adding LiDAR features (zYyJ, KFME, vyfW).

We disentangled the sources of AutoWorld's gains by comparing a trajectory-only model, a model conditioned on observed LiDAR and AutoWorld conditioned on world-model forecasts (3 seeds per variant, DPP disabled). Under high occlusion, observed geometry lowers the collision rate from 58.01% to 42.68%, and learned forecasting lowers it further to 15.35%.

2. Motion-aware supervision recovers most of the value of labels (zYyJ).

We compared our motion-aware objective with standard rectified flow, with and without semantic labels, across occlusion levels. It lowers collisions by 25–41% relative to standard rectified flow and recovers 46–80% of the gain from semantic labels, without annotations.

3. AutoWorld benefits from semantics, not only geometry (p11Q, zYyJ).

We evaluated the effect of adding semantic labels to world-model training. With the same objective, labels lower collisions by 45–56% at every occlusion level. AutoWorld's motion model already receives map elements, traffic-light states and object types, and cameras can provide further semantics without labels, which makes multimodal world models a direct extension. We have scoped our claims to LiDAR grounding.

4. AutoWorld reduces the collision rate by ∼70% relative to Nexus, at 1.48× its runtime. (p11Q, zYyJ, KFME).

We measured end-to-end latency against Nexus on the same GPU and analyzed how DPP's cost scales with the simulation horizon. AutoWorld takes 146 ms per rollout vs. 99 ms for Nexus (1.48×) while lowering collisions by 70%. DPP guidance adds no sampling steps and accounts for 7.3% of runtime, a share that stays constant as the horizon grows. Training takes 193 A100 GPU-hours.

5. DPP improves diversity over a simpler alternative at negligible extra cost (zYyJ, KFME).

We replaced DPP with pairwise repulsive guidance, using the same quality weights, schedule and rollout budget, in the safety-critical generation experiment. DPP improves diversity by 31–45% and achieves better realism. Following KFME's suggestion, we have reframed it as diversity-encouraging sampling method, whose strength is tuned once per dataset on held-out validation data.

AuthorsOpenReview ↗

Summary of Rebuttal: Reviewer Feedback, Additional Results, and Discussion Status (2/2)

6. AutoWorld achieves a better controllability–realism trade-off than Nexus at every guidance strength (zYyJ).

We swept the safety-critical guidance strength from 0.5× to 2× for Nexus and for AutoWorld with and without DPP. AutoWorld+DPP's realism stays within 0.253–0.313 (lower is better), better than Nexus's best (0.322). Nexus degrades to 0.569, with up to 46.59% non-target collisions vs. at most 12.52% for AutoWorld+DPP. At the default strength, DPP improves controllability and realism together (target collisions 34.56% → 41.84%, realism 0.289 → 0.274).

7. AutoWorld is stable over longer rollouts (vyfW, p11Q).

We evaluated closed-loop simulation at the benchmark horizon and at twice that horizon, rolling the world model out autoregressively. Doubling the horizon raises AutoWorld's collision rate by 3.34 points, vs. 17.7 points for the trajectory-only baseline.

8. AutoWorld's world model transfers across LiDAR rigs (zYyJ).

We trained the world model on Waymo, kept it frozen, and trained only the motion model on nuPlan, whose LiDAR configuration differs. It retains 84% of the gain from a world model trained on nuPlan itself.

Key clarifications. The reactiveness test (Sec. 4.3) is fully closed-loop, with receding-horizon simulation over 8 s at 1 Hz, not a single-step interaction (p11Q). The world model is trained on real LiDAR returns with visibility masks, treats unobserved regions as unknown and never uses agent tracks, so tracking errors do not reach it. BEV line-of-sight ray casting is used only to assign occlusion levels for evaluation (vyfW, p11Q). We follow each benchmark's established realism protocol and do not compare scores across benchmarks. nuPlan measures realism with a distribution-based metric and WOSAC with a likelihood-based one, so evaluating on both assesses AutoWorld's realism from two complementary perspectives (p11Q). Video world models such as GAIA-1, Vista and DriveDreamer do not output per-agent trajectories, so they are not direct baselines (zYyJ).

All new results and clarifications have been incorporated into the paper and its appended appendix, and they have strengthened our work.

Reviewer p11Q, the only reviewer currently below the acceptance threshold, has not yet responded to our rebuttal or our follow-up. Their main concerns (compute cost, closed-loop reactiveness, sensor imperfections, realism metrics and the role of semantics) are addressed by items 3, 4 and 7 and by the clarifications above. We would appreciate the AC considering these responses and new results in the final assessment.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.