- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies which world representation best supports world-action models for driving. It builds on a common architecture, a Mixture-of-Transformers diffusion model, an action expert and an RAE-style wide head, trained with flow matching. By changing the attention mask, the same model acts as an inverse dynamics model, an action-conditioned world model or a WAM. The world is represented by the features of one of nine frozen encoders, ranging from self-supervised and language-aligned features to geometry models, image autoencoders and a BEV model. The WM is pretrained on nuPlan and fine-tuned on NAVSIM navtrain and the WAM is initialized from the WM. Three tests are performed to compare the representations. The IDM recovers the trajectory from real frames. The WM is scored by passing its predicted latents to the frozen IDM of the same representation and comparing the recovered trajectory with the conditioning one. The WAM is scored by NAVSIM planning. In all three tests DINOv3 and V-JEPA 2.1 perform best. The paper then compares ways for world and action prediction to interact and proposes execute-then-imagine. In ETI, world tokens attend to the planned actions during training and only actions are generated at inference. ReWAM reports 91.6 PDMS on NAVSIM v1 and 91.7 EPDMS on NAVSIM v2, the best among the listed methods and shows zero-shot transfer to Bench2Drive, which is based on CARLA.
Critical strengths
- S1. The paper runs a controlled and fair comparison of nine world representations. It keeps the architecture, data, conditioning and training schedule fixed and varies only the frozen encoder, across three tasks. The trends agree across all proposed tasks and shows that FID and FVD do not follow planning quality, which supports evaluating world models without decoding. The study gives practical guidance on which representation to use for driving world models. This is the first clean study that compares different encoder representations for world modeling and evaluates on the actual downstream task of planning.
- S2. With one front camera and 698M parameters, ReWAM reaches PDMS and navhard EPDMS is higher than that of WAMs with much more parameters. Further, the proposed ETI-cache keeps the perfroamance and inference speed, compared to the action-only baseline without world supervision. This suggests shows with evidence that world model training can improve planning without generating future states at inference.
- S3. The model transfers zero-shot to Bench2Drive, with a lowest average L2 error and collision rate than the other WAMs trained only on NAVSIM.
Critical weaknesses
- W1. The world model may recall nuPlan locations rather than predict from its input. Pretraining runs 60k steps at batch 128 on the nuPlan training split, which by my estimate is roughly 5 to 15 passes over heavily overlapping front-camera clips. Nearly all evaluation data (navtest, navhard, nuPlan val) come from the same four cities. The supplementary video consistent_world_action.mp4 points to recall. In the turning examples, the generated frames after the turn show the same buildings, textures and structures as the real video, although these are not visible in the 1.5 s of observed history and the model receives no map. The generated ego motion also closely follows the real one. This raises the question whether the model can actually generate counterfactual futures or only reproduces drives it has seen during training. The paper should state the split and logs of all shown scenes, evaluate on locations absent from pretraining and report a nearest-neighbour check of generated frames against training frames to show how much of the world modeling is recall.
- W2. The world-model test mostly measures the IDM and cannot detect recall. The paper evaluates each WM by passing its predicted latents to the IDM of the same representation and checking whether the IDM recovers the given trajectory. Each representation is therefore judged by its own IDM and even a perfect WM cannot score better than its IDM does on real frames. For DINOv3 the error is 0.752 m on real frames and 0.761 m on predicted latents. For BEV it is 1.151 m and 1.194 m. The WM adds only 0.009 to 0.043 m. The ranking in Table 2 and the correlation of r = 0.999 between IDM and WM in Fig. 5 are therefore largely built into the test. A WM that simply recalls the real future would also pass this test. This also leaves open question whether DINOv3 and V-JEPA 2.1 lead because they model dynamics better or because their features make familiar places easier to recall. The authors should report the error each WM adds over its own IDM and also show using one shared action estimator for all WMs. They should also report the zero-shot ranking of all nine representations on Bench2Drive and compare the ranking on navtest scenes at locations seen and unseen during pretraining.
- W3. No quantitative evidence for long-horizon prediction. The paper states that ReWAM maintains scene consistency over one-minute rollouts, shows the most plausible world dynamics and that foundation representations are a promising basis for long-horizon driving simulators. These claims rest only on one figure and one supplementary video. The number of scenes and the rollout procedure are not stated and no metric is reported. The claim should be evaluated on many held-out scenes with metrics per time window (for example FVD, frozen-IDM action error on the rolled-out latents and feature distance to the ground truth) applied in the same way to the baselines and to ReWAM with other representations.
Questions for authors
- Q1 (W1). How many nuPlan clips are used for WM pretraining, with what sampling stride and how many epochs does this correspond to? Are the navtest logs disjoint from the nuPlan logs used for pretraining?
- Q2 (W1). Which split and logs do the scenes in Fig. 7 and the supplementary videos come from? Could you quantify how often generated content after turns matches unseen ground truth, evaluate on locations absent from pretraining and report a nearest-neighbour check of generated frames against training frames?
- Q3 (W2). Could you report the error each WM adds over its own IDM on real frames, or evaluate all WMs with one shared action estimator?
- Q4 (W2). Could you report the zero-shot ranking of all nine representations on Bench2Drive and the ranking on navtest scenes at locations seen and unseen during pretraining?
- Q5 (W3). For the one-minute rollouts, how many frames does each autoregressive step add, does every step use only predicted frames as context, and which actions, ego states and commands condition each step after the observed context ends? How many scenes were tested?
- Q6. In ETI-cache, the action expert first predicts a trajectory and the world expert then predicts the future from that trajectory. The world target is the real future, which was produced by the real trajectory. So the world loss may mostly pull the predicted trajectory toward the real one, which works like a second imitation loss rather than learning world dynamics. Table 14 shows that this gradient helps, but not why. Could you compare ETI-cache with a simpler variant that trains the rolled-out trajectory directly toward the real trajectory, with no world prediction?
10 comments
Response to Reviewer dwyQ (Part 2/5)
1. Location overlap and potential recall of training scenes (continued)
W1: Generated futures may reproduce previously observed drives.
The turning examples in
consistent_world_action.mp4appear to reproduce buildings and structures that are not visible in the input. Establish whether the model can predict plausible futures at locations absent from training, rather than reproduce familiar recordings.
We appreciate the request for additional visual evidence. We add frame-level comparisons with ground truth on multiple held-out Boston scenes using the retrained ReWAM model. These locations are absent from training. The examples are presented in Figure 10 of the revised paper and the new supplementary video unseen_boston_cases_multi_seed.mp4.
ReWAM generates plausible futures at these unseen locations, with clear differences from the recorded ground truth. With the input fixed, changing only the sampling noise seed produces different future predictions. These results support the model's ability to generate diverse futures at unseen locations, rather than simply replay familiar training drives.
We have added these visualizations to Appendix C.4 (Figure 10) of the revised paper and the new video to the supplementary materials.
Q1 & W1: Pretraining clips, sampling stride, effective epochs, and log overlap.
State how many nuPlan clips are used for WM pretraining, the sampling stride, and the corresponding effective training epochs. Confirm whether navtest logs are disjoint from the pretraining logs.
For the original split, WM pretraining uses only nuPlan train, comprising 15,949 source sequences from 1,085 logs. We randomly sample 12-frame windows (4 history + 8 future) at 2 Hz, with a within-window frame stride of 1. For a rough estimate, counting all possible window starts gives approximately 1.04 million candidate windows. With 60k training steps and a global batch size of 128, this corresponds to approximately 7.4 window-equivalent epochs.
None of the 136 navtest logs overlap with the logs used for WM pretraining. Downstream training uses navtrain, and navtest is reserved for evaluation. The original-split statistics are summarized below. The city-disjoint experiments above additionally remove location overlap.
Original-split data and log overlap.
| Data split | Logs | Source sequences / scene anchors | Logs overlapping navtest |
|---|---|---|---|
| nuPlan train | 1,085 | 15,949 sequences | 0 |
| nuPlan val | 209 | 2,315 sequences | 0 |
| nuPlan test | 147 | 1,972 sequences | 136 |
| NAVSIM navtrain | 1,192 | 103,288 anchors | 0 |
| NAVSIM navtest | 136 | 12,146 anchors | — |
We have clarified that training and evaluation logs are disjoint in Appendix A.2 of the revised paper.
Q2 & W1: Sources of the visualized scenes and checks for location recall.
Identify the splits and logs of the scenes in Figure 7 and the supplementary videos. Quantify how often generated post-turn content matches ground truth unseen in the input, evaluate locations absent from pretraining, and check nearest neighbours of generated frames in the training data.
Fig. 7 and long_horizon.mp4 use three Boston logs from navtest (nuPlan test):
2021.08.30.14.54.34_veh-40_00334_00419
2021.09.29.18.19.40_veh-28_00844_01218
2021.08.30.14.54.34_veh-40_00439_00835
The 60-second reference sequences are extracted continuously from these test logs at 2 Hz.
Fig. 6 and consistent_world_action.mp4 use 10 logs from navtest (nuPlan test):
2021.09.29.14.44.26_veh-28_00528_00992
2021.09.29.18.19.40_veh-28_00844_01218
2021.09.16.15.12.03_veh-42_01037_01434
2021.06.03.17.06.58_veh-35_02571_02742
2021.08.30.16.16.44_veh-40_00779_01088
2021.09.29.14.44.26_veh-28_00337_00504
2021.09.29.15.23.04_veh-28_00601_00802
2021.09.29.19.02.14_veh-28_02911_03005
2021.09.16.19.12.04_veh-42_01221_01380
2021.09.16.14.39.34_veh-42_00297_00935
We found post-turn content matches difficult to quantify consistently at scale, as judging agreement with ground-truth content involves subjective decisions without a uniform matching criterion. In our exploratory nearest-neighbour check, feature retrieval over millions of training frames was time-consuming, and the retrieved images generally showed little visual correspondence to the generated frames, making the check difficult to scale and interpret. However, we believe the held-out-city experiment described above directly addresses the underlying location-recall concern: Boston is excluded from all stages of our nuPlan/NAVSIM training, and evaluation uses only Boston scenes.
We have included the held-out-city experiments and visualizations addressing this concern in Appendix C.4 of the revised paper.
Response to Reviewer dwyQ (Part 3/5)
2. WM evaluation with one shared action estimator
Q3 & W2: Representation-specific IDM errors may dominate WM evaluation.
WM consistency errors closely track each representation's IDM errors, raising concern that the ranking and the near-perfect IDM–WM correlation reflect evaluator differences. Report the error each WM adds relative to its own IDM on real futures, or evaluate all WMs with one shared action estimator.
We appreciate this suggestion. We follow the suggestion to use one shared action estimator and reevaluate the original representation variants. This removes differences in IDM action-recovery ability from the WM comparison.
Specifically, to use the same IDM across representations, we first decode their predicted world latents into images and re-encode them with one frozen DINOv3 encoder. This maps all predictions into a common feature space, where one frozen DINOv3-based IDM recovers the actions. We apply the same decode/re-encode path to the DINOv3 variant as well. BEV is excluded because its spatial features are not aligned with image pixels and cannot use this image-based conversion.
To test whether representation differences in reconstruction quality along the decode/re-encode path affect the shared IDM, we also evaluate a reconstruction reference for each representation. We encode and decode real future images using that representation, then apply the same shared IDM estimator. Table R4 compares the resulting reference errors with errors on generated futures, both measured against the corresponding conditioning actions.
Table R4. WM evaluation with a shared DINOv3-based IDM. Overall action error (m) ↓, using the definition in manuscript Table 2.
| Representation | Reconstruction-reference error ↓ | Generated-future error ↓ |
|---|---|---|
| DINOv3 | 0.768 | 0.774 |
| V-JEPA 2.1 | 0.773 | 0.781 |
| SigLIP 2 | 0.779 | 0.806 |
| CLIP | 0.781 | 0.814 |
| DA3 | 0.770 | 0.803 |
| MoGe-2 | 0.761 | 0.800 |
| SD-VAE | 0.784 | 1.365 |
| DCAE | 0.798 | 1.466 |
As shown in Table R4, the similar reconstruction-reference errors suggest that the DINOv3-based IDM is relatively insensitive to reconstruction differences along the decode/re-encode path. Thus it is suitable as a shared action estimator across representations. Under the shared evaluation protocol, image/video self-supervised representations achieve the lowest errors, and foundation representations substantially outperform SD-VAE and DCAE. These trends agree with our original results and show that the representation advantages are not solely driven by differences among their individual IDMs.
We have added this shared-IDM evaluation and its reconstruction-reference results to Appendix B.6 of the revised paper.
3. Zero-shot representation ranking on Bench2Drive
Q4 & W2-1: Generalization of representation rankings beyond the pretraining domain.
Report the zero-shot ranking of all nine representations on Bench2Drive to test whether the original representation trends persist outside nuPlan/NAVSIM and are not explained by familiarity with training locations.
Thank you for recommending this zero-shot comparison. We evaluate all nine NAVSIM-trained WAM variants on Bench2Drive without target-domain fine-tuning, using the original representation-comparison settings. Following manuscript Table 7, we report L2 error and collision metrics at 1, 2, and 3 seconds, together with their averages. Table R5 compares the representations under this new domain.
Table R5. Zero-shot planning across representations on Bench2Drive. Lower is better for all metrics.
| Metric | DINOv3 | V-JEPA 2.1 | SigLIP 2 | CLIP | DA3 | MoGe-2 | SD-VAE | DCAE | BEVFormer |
|---|---|---|---|---|---|---|---|---|---|
| L2 at 1 s (m) | 0.55 | 0.54 | 0.55 | 0.57 | 0.60 | 0.55 | 0.83 | 0.80 | 1.07 |
| L2 at 2 s (m) | 1.23 | 1.22 | 1.29 | 1.33 | 1.30 | 1.27 | 1.49 | 1.58 | 2.69 |
| L2 at 3 s (m) | 2.12 | 2.14 | 2.26 | 2.30 | 2.14 | 2.17 | 2.40 | 2.43 | 2.81 |
| Avg. L2 (m) | 1.30 | 1.30 | 1.37 | 1.40 | 1.35 | 1.33 | 1.58 | 1.60 | 2.19 |
| Collision at 1 s (%) | 0.95 | 0.83 | 0.94 | 0.94 | 0.99 | 0.98 | 1.90 | 2.14 | 2.01 |
| Collision at 2 s (%) | 1.56 | 1.41 | 1.80 | 1.84 | 1.73 | 1.57 | 2.20 | 2.62 | 3.04 |
| Collision at 3 s (%) | 2.42 | 2.15 | 2.94 | 2.88 | 2.66 | 2.49 | 3.08 | 3.75 | 3.82 |
| Avg. collision (%) | 1.64 | 1.46 | 1.89 | 1.89 | 1.79 | 1.68 | 2.39 | 2.84 | 2.96 |
As shown in Table R5, image/video self-supervised representations remain strongest overall on Bench2Drive. DINOv3 and V-JEPA 2.1 tie for the lowest average L2 error, while V-JEPA 2.1 achieves the lowest average collision rate. Foundation representations also outperform reconstruction and BEV alternatives on both average metrics. These results indicate that the main representation trends hold beyond nuPlan/NAVSIM.
We have reported the zero-shot representation comparison on Bench2Drive in Appendix B.5 of the revised paper.
Response to Reviewer dwyQ (Part 1/5)
We thank the reviewer for the constructive feedback. Below, we present additional experiments and clarify our training data, evaluation protocols, and interaction design to address the concerns.
1. Location overlap and potential recall of training scenes
Thank you for raising this concern. The standard nuPlan/NAVSIM splits separate recording logs but do not enforce geographic separation. Different training and test logs can revisit the same roads and static backgrounds. Our original experiments follow this protocol for comparability with prior methods such as Epona and DriveVLA-W0. Thus, unseen test recordings may still contain static backgrounds encountered during training, which can trigger location-specific recall at inference.
To address this concern, we introduce a city-disjoint split for nuPlan/NAVSIM, using three cities for training and the remaining city for evaluation. This eliminates location overlap and rules out recall of evaluation locations from the training recordings.
Original and city-disjoint data splits. The new splits are shown in bold.
We rerun the experiments below under this split to assess representation rankings and future prediction at locations absent from training.
We have added the city-disjoint evaluation setting and experiments to Appendix C.4 of the revised paper.
Thank you for suggesting this comparison. We retrain IDM, WM, and WAM under the city-disjoint split, keeping all other settings unchanged. Given the limited time for the rebuttal, we select the strongest representative from each representation family: DINOv3, SigLIP 2, DA3, SD-VAE, and BEVFormer. Tables R1–R3 report their results on the held-out city.
Table R1. IDM action recovery on the held-out-city navtest split.
Table R2. WM world-action consistency on the held-out-city navtest split.
Table R3. WAM planning on the held-out-city NAVSIM v1, v2, and navhard.
Tables R1–R3 show that the original representation trends persist at held-out locations: DINOv3 remains strongest, and foundation representations outperform reconstruction and BEV alternatives. DINOv3's advantage generally becomes larger, with aligned trends across action recovery, world-action consistency, and planning. These results support our original findings and show that self-supervised representations retain their advantage at locations unseen during training. Their advantage therefore cannot be explained simply by making familiar places easier to recall.
We have included these held-out-city representation comparisons in Appendix C.4 of the revised paper.