acceptodds
Under review as a conference paper at ICLR 2027

ReWAM: A Unified Framework for Representation-Centric Analysis of World-Action Modeling

Abstract

World-action models (WAMs) learn future scene dynamics alongside actions for autonomous driving, yet how world representations affect planning remains unclear. We propose ReWAM, a unified framework for studying how world representations shape planning. Under a common architecture, ReWAM instantiates inverse dynamics model, world model, and world-action model across nine representations spanning VAE latents, visual foundation features, and driving-specific BEV features, while making even implicit foundation latents visually inspectable as videos. We evaluate these models progressively through action recovery, appearance-agnostic world-action consistency, and planning, revealing aligned representation trends and an advantage for visual foundation representations. Within the same framework, we further compare world-action interaction paradigms and introduce a new method that directs world-prediction supervision to action generation during training while retaining action-only inference. ReWAM achieves strong planning performance, reaching 91.6 PDMS on navtest and 39.3 EPDMS on navhard. Its efficient variant retains 91.6 PDMS on navtest at 59 ms, 3.6× faster than inference with future-world generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, World-Action Model, Autonomous Driving, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dwyQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies which world representation best supports world-action models for driving. It builds on a common architecture, a Mixture-of-Transformers diffusion model, an action expert and an RAE-style wide head, trained with flow matching. By changing the attention mask, the same model acts as an inverse dynamics model, an action-conditioned world model or a WAM. The world is represented by the features of one of nine frozen encoders, ranging from self-supervised and language-aligned features to geometry models, image autoencoders and a BEV model. The WM is pretrained on nuPlan and fine-tuned on NAVSIM navtrain and the WAM is initialized from the WM. Three tests are performed to compare the representations. The IDM recovers the trajectory from real frames. The WM is scored by passing its predicted latents to the frozen IDM of the same representation and comparing the recovered trajectory with the conditioning one. The WAM is scored by NAVSIM planning. In all three tests DINOv3 and V-JEPA 2.1 perform best. The paper then compares ways for world and action prediction to interact and proposes execute-then-imagine. In ETI, world tokens attend to the planned actions during training and only actions are generated at inference. ReWAM reports 91.6 PDMS on NAVSIM v1 and 91.7 EPDMS on NAVSIM v2, the best among the listed methods and shows zero-shot transfer to Bench2Drive, which is based on CARLA.

Critical strengths
  • S1. The paper runs a controlled and fair comparison of nine world representations. It keeps the architecture, data, conditioning and training schedule fixed and varies only the frozen encoder, across three tasks. The trends agree across all proposed tasks and shows that FID and FVD do not follow planning quality, which supports evaluating world models without decoding. The study gives practical guidance on which representation to use for driving world models. This is the first clean study that compares different encoder representations for world modeling and evaluates on the actual downstream task of planning.
  • S2. With one front camera and 698M parameters, ReWAM reaches PDMS and navhard EPDMS is higher than that of WAMs with much more parameters. Further, the proposed ETI-cache keeps the perfroamance and inference speed, compared to the action-only baseline without world supervision. This suggests shows with evidence that world model training can improve planning without generating future states at inference.
  • S3. The model transfers zero-shot to Bench2Drive, with a lowest average L2 error and collision rate than the other WAMs trained only on NAVSIM.
Critical weaknesses
  • W1. The world model may recall nuPlan locations rather than predict from its input. Pretraining runs 60k steps at batch 128 on the nuPlan training split, which by my estimate is roughly 5 to 15 passes over heavily overlapping front-camera clips. Nearly all evaluation data (navtest, navhard, nuPlan val) come from the same four cities. The supplementary video consistent_world_action.mp4 points to recall. In the turning examples, the generated frames after the turn show the same buildings, textures and structures as the real video, although these are not visible in the 1.5 s of observed history and the model receives no map. The generated ego motion also closely follows the real one. This raises the question whether the model can actually generate counterfactual futures or only reproduces drives it has seen during training. The paper should state the split and logs of all shown scenes, evaluate on locations absent from pretraining and report a nearest-neighbour check of generated frames against training frames to show how much of the world modeling is recall.
  • W2. The world-model test mostly measures the IDM and cannot detect recall. The paper evaluates each WM by passing its predicted latents to the IDM of the same representation and checking whether the IDM recovers the given trajectory. Each representation is therefore judged by its own IDM and even a perfect WM cannot score better than its IDM does on real frames. For DINOv3 the error is 0.752 m on real frames and 0.761 m on predicted latents. For BEV it is 1.151 m and 1.194 m. The WM adds only 0.009 to 0.043 m. The ranking in Table 2 and the correlation of r = 0.999 between IDM and WM in Fig. 5 are therefore largely built into the test. A WM that simply recalls the real future would also pass this test. This also leaves open question whether DINOv3 and V-JEPA 2.1 lead because they model dynamics better or because their features make familiar places easier to recall. The authors should report the error each WM adds over its own IDM and also show using one shared action estimator for all WMs. They should also report the zero-shot ranking of all nine representations on Bench2Drive and compare the ranking on navtest scenes at locations seen and unseen during pretraining.
  • W3. No quantitative evidence for long-horizon prediction. The paper states that ReWAM maintains scene consistency over one-minute rollouts, shows the most plausible world dynamics and that foundation representations are a promising basis for long-horizon driving simulators. These claims rest only on one figure and one supplementary video. The number of scenes and the rollout procedure are not stated and no metric is reported. The claim should be evaluated on many held-out scenes with metrics per time window (for example FVD, frozen-IDM action error on the rolled-out latents and feature distance to the ground truth) applied in the same way to the baselines and to ReWAM with other representations.
Questions for authors
  • Q1 (W1). How many nuPlan clips are used for WM pretraining, with what sampling stride and how many epochs does this correspond to? Are the navtest logs disjoint from the nuPlan logs used for pretraining?
  • Q2 (W1). Which split and logs do the scenes in Fig. 7 and the supplementary videos come from? Could you quantify how often generated content after turns matches unseen ground truth, evaluate on locations absent from pretraining and report a nearest-neighbour check of generated frames against training frames?
  • Q3 (W2). Could you report the error each WM adds over its own IDM on real frames, or evaluate all WMs with one shared action estimator?
  • Q4 (W2). Could you report the zero-shot ranking of all nine representations on Bench2Drive and the ranking on navtest scenes at locations seen and unseen during pretraining?
  • Q5 (W3). For the one-minute rollouts, how many frames does each autoregressive step add, does every step use only predicted frames as context, and which actions, ego states and commands condition each step after the observed context ends? How many scenes were tested?
  • Q6. In ETI-cache, the action expert first predicts a trajectory and the world expert then predicts the future from that trajectory. The world target is the real future, which was produced by the real trajectory. So the world loss may mostly pull the predicted trajectory toward the real one, which works like a second imitation loss rather than learning world dynamics. Table 14 shows that this gradient helps, but not why. Could you compare ETI-cache with a simpler variant that trains the rolled-out trajectory directly toward the real trajectory, with no world prediction?
10 comments
AuthorsOpenReview ↗

Response to Reviewer dwyQ (Part 1/5)

We thank the reviewer for the constructive feedback. Below, we present additional experiments and clarify our training data, evaluation protocols, and interaction design to address the concerns.

1. Location overlap and potential recall of training scenes

W1 & W2 (location-related): Location recall versus input-conditioned world prediction.

Training and evaluation recordings come from the same four cities and may revisit the same locations. Generated post-turn content resembles the recorded future despite being absent from the observed history. The reviewer questions whether the model recalls familiar scenes and whether this familiarity explains the stronger results of DINOv3 and V-JEPA 2.1.

Thank you for raising this concern. The standard nuPlan/NAVSIM splits separate recording logs but do not enforce geographic separation. Different training and test logs can revisit the same roads and static backgrounds. Our original experiments follow this protocol for comparability with prior methods such as Epona and DriveVLA-W0. Thus, unseen test recordings may still contain static backgrounds encountered during training, which can trigger location-specific recall at inference.

To address this concern, we introduce a city-disjoint split for nuPlan/NAVSIM, using three cities for training and the remaining city for evaluation. This eliminates location overlap and rules out recall of evaluation locations from the training recordings.

Original and city-disjoint data splits. The new splits are shown in bold.

Data splitVersionLas VegasBostonPittsburghSingaporeTotal
nuPlan trainOriginal12,4601,6491,34449615,949
nuPlan trainCity-disjoint12,46001,34449614,300
nuPlan testOriginal7546943182061,972
nuPlan testCity-disjoint069400694
NAVSIM navtrainOriginal67,68312,95410,82111,830103,288
NAVSIM navtrainCity-disjoint67,683010,82111,83090,334
NAVSIM navtestOriginal4,0643,7312,4371,91412,146
NAVSIM navtestCity-disjoint03,731003,731

We rerun the experiments below under this split to assess representation rankings and future prediction at locations absent from training.

We have added the city-disjoint evaluation setting and experiments to Appendix C.4 of the revised paper.

Q4 & W2-2: Representation rankings at locations seen and unseen during pretraining.

Compare representation rankings on navtest at locations seen versus unseen during pretraining, to determine whether the original trends depend on location familiarity.

Thank you for suggesting this comparison. We retrain IDM, WM, and WAM under the city-disjoint split, keeping all other settings unchanged. Given the limited time for the rebuttal, we select the strongest representative from each representation family: DINOv3, SigLIP 2, DA3, SD-VAE, and BEVFormer. Tables R1–R3 report their results on the held-out city.

Table R1. IDM action recovery on the held-out-city navtest split.

MetricDINOv3SigLIP 2DA3SD-VAEBEVFormer
ADE (m) ↓0.5830.8140.7941.1011.220
FDE (m) ↓0.9421.3151.1971.3551.391
Overall (m) ↓0.7631.0650.9961.2281.306

Table R2. WM world-action consistency on the held-out-city navtest split.

MetricDINOv3SigLIP 2DA3SD-VAEBEVFormer
ADE (m) ↓0.6100.8300.8131.1711.286
FDE (m) ↓0.9731.3541.2091.3831.450
Overall (m) ↓0.7921.0921.0111.2771.368

Table R3. WAM planning on the held-out-city NAVSIM v1, v2, and navhard.

MetricDINOv3SigLIP 2DA3SD-VAEBEVFormer
PDMS (v1) ↑89.988.388.886.185.9
EPDMS (v2) ↑90.288.489.086.586.3
EPDMS (navhard) ↑37.631.933.227.727.4

Tables R1–R3 show that the original representation trends persist at held-out locations: DINOv3 remains strongest, and foundation representations outperform reconstruction and BEV alternatives. DINOv3's advantage generally becomes larger, with aligned trends across action recovery, world-action consistency, and planning. These results support our original findings and show that self-supervised representations retain their advantage at locations unseen during training. Their advantage therefore cannot be explained simply by making familiar places easier to recall.

We have included these held-out-city representation comparisons in Appendix C.4 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer dwyQ (Part 2/5)

1. Location overlap and potential recall of training scenes (continued)

W1: Generated futures may reproduce previously observed drives.

The turning examples in consistent_world_action.mp4 appear to reproduce buildings and structures that are not visible in the input. Establish whether the model can predict plausible futures at locations absent from training, rather than reproduce familiar recordings.

We appreciate the request for additional visual evidence. We add frame-level comparisons with ground truth on multiple held-out Boston scenes using the retrained ReWAM model. These locations are absent from training. The examples are presented in Figure 10 of the revised paper and the new supplementary video unseen_boston_cases_multi_seed.mp4.

ReWAM generates plausible futures at these unseen locations, with clear differences from the recorded ground truth. With the input fixed, changing only the sampling noise seed produces different future predictions. These results support the model's ability to generate diverse futures at unseen locations, rather than simply replay familiar training drives.

We have added these visualizations to Appendix C.4 (Figure 10) of the revised paper and the new video to the supplementary materials.

Q1 & W1: Pretraining clips, sampling stride, effective epochs, and log overlap.

State how many nuPlan clips are used for WM pretraining, the sampling stride, and the corresponding effective training epochs. Confirm whether navtest logs are disjoint from the pretraining logs.

For the original split, WM pretraining uses only nuPlan train, comprising 15,949 source sequences from 1,085 logs. We randomly sample 12-frame windows (4 history + 8 future) at 2 Hz, with a within-window frame stride of 1. For a rough estimate, counting all possible window starts gives approximately 1.04 million candidate windows. With 60k training steps and a global batch size of 128, this corresponds to approximately 7.4 window-equivalent epochs.

None of the 136 navtest logs overlap with the logs used for WM pretraining. Downstream training uses navtrain, and navtest is reserved for evaluation. The original-split statistics are summarized below. The city-disjoint experiments above additionally remove location overlap.

Original-split data and log overlap.

Data splitLogsSource sequences / scene anchorsLogs overlapping navtest
nuPlan train1,08515,949 sequences0
nuPlan val2092,315 sequences0
nuPlan test1471,972 sequences136
NAVSIM navtrain1,192103,288 anchors0
NAVSIM navtest13612,146 anchors—

We have clarified that training and evaluation logs are disjoint in Appendix A.2 of the revised paper.

Q2 & W1: Sources of the visualized scenes and checks for location recall.

Identify the splits and logs of the scenes in Figure 7 and the supplementary videos. Quantify how often generated post-turn content matches ground truth unseen in the input, evaluate locations absent from pretraining, and check nearest neighbours of generated frames in the training data.

Fig. 7 and long_horizon.mp4 use three Boston logs from navtest (nuPlan test):

2021.08.30.14.54.34_veh-40_00334_00419
2021.09.29.18.19.40_veh-28_00844_01218
2021.08.30.14.54.34_veh-40_00439_00835

The 60-second reference sequences are extracted continuously from these test logs at 2 Hz.

Fig. 6 and consistent_world_action.mp4 use 10 logs from navtest (nuPlan test):

2021.09.29.14.44.26_veh-28_00528_00992
2021.09.29.18.19.40_veh-28_00844_01218
2021.09.16.15.12.03_veh-42_01037_01434
2021.06.03.17.06.58_veh-35_02571_02742
2021.08.30.16.16.44_veh-40_00779_01088
2021.09.29.14.44.26_veh-28_00337_00504
2021.09.29.15.23.04_veh-28_00601_00802
2021.09.29.19.02.14_veh-28_02911_03005
2021.09.16.19.12.04_veh-42_01221_01380
2021.09.16.14.39.34_veh-42_00297_00935

We found post-turn content matches difficult to quantify consistently at scale, as judging agreement with ground-truth content involves subjective decisions without a uniform matching criterion. In our exploratory nearest-neighbour check, feature retrieval over millions of training frames was time-consuming, and the retrieved images generally showed little visual correspondence to the generated frames, making the check difficult to scale and interpret. However, we believe the held-out-city experiment described above directly addresses the underlying location-recall concern: Boston is excluded from all stages of our nuPlan/NAVSIM training, and evaluation uses only Boston scenes.

We have included the held-out-city experiments and visualizations addressing this concern in Appendix C.4 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer dwyQ (Part 3/5)

2. WM evaluation with one shared action estimator

Q3 & W2: Representation-specific IDM errors may dominate WM evaluation.

WM consistency errors closely track each representation's IDM errors, raising concern that the ranking and the near-perfect IDM–WM correlation reflect evaluator differences. Report the error each WM adds relative to its own IDM on real futures, or evaluate all WMs with one shared action estimator.

We appreciate this suggestion. We follow the suggestion to use one shared action estimator and reevaluate the original representation variants. This removes differences in IDM action-recovery ability from the WM comparison.

Specifically, to use the same IDM across representations, we first decode their predicted world latents into images and re-encode them with one frozen DINOv3 encoder. This maps all predictions into a common feature space, where one frozen DINOv3-based IDM recovers the actions. We apply the same decode/re-encode path to the DINOv3 variant as well. BEV is excluded because its spatial features are not aligned with image pixels and cannot use this image-based conversion.

To test whether representation differences in reconstruction quality along the decode/re-encode path affect the shared IDM, we also evaluate a reconstruction reference for each representation. We encode and decode real future images using that representation, then apply the same shared IDM estimator. Table R4 compares the resulting reference errors with errors on generated futures, both measured against the corresponding conditioning actions.

Table R4. WM evaluation with a shared DINOv3-based IDM. Overall action error (m) ↓, using the definition in manuscript Table 2.

RepresentationReconstruction-reference error ↓Generated-future error ↓
DINOv30.7680.774
V-JEPA 2.10.7730.781
SigLIP 20.7790.806
CLIP0.7810.814
DA30.7700.803
MoGe-20.7610.800
SD-VAE0.7841.365
DCAE0.7981.466

As shown in Table R4, the similar reconstruction-reference errors suggest that the DINOv3-based IDM is relatively insensitive to reconstruction differences along the decode/re-encode path. Thus it is suitable as a shared action estimator across representations. Under the shared evaluation protocol, image/video self-supervised representations achieve the lowest errors, and foundation representations substantially outperform SD-VAE and DCAE. These trends agree with our original results and show that the representation advantages are not solely driven by differences among their individual IDMs.

We have added this shared-IDM evaluation and its reconstruction-reference results to Appendix B.6 of the revised paper.

3. Zero-shot representation ranking on Bench2Drive

Q4 & W2-1: Generalization of representation rankings beyond the pretraining domain.

Report the zero-shot ranking of all nine representations on Bench2Drive to test whether the original representation trends persist outside nuPlan/NAVSIM and are not explained by familiarity with training locations.

Thank you for recommending this zero-shot comparison. We evaluate all nine NAVSIM-trained WAM variants on Bench2Drive without target-domain fine-tuning, using the original representation-comparison settings. Following manuscript Table 7, we report L2 error and collision metrics at 1, 2, and 3 seconds, together with their averages. Table R5 compares the representations under this new domain.

Table R5. Zero-shot planning across representations on Bench2Drive. Lower is better for all metrics.

MetricDINOv3V-JEPA 2.1SigLIP 2CLIPDA3MoGe-2SD-VAEDCAEBEVFormer
L2 at 1 s (m)0.550.540.550.570.600.550.830.801.07
L2 at 2 s (m)1.231.221.291.331.301.271.491.582.69
L2 at 3 s (m)2.122.142.262.302.142.172.402.432.81
Avg. L2 (m)1.301.301.371.401.351.331.581.602.19
Collision at 1 s (%)0.950.830.940.940.990.981.902.142.01
Collision at 2 s (%)1.561.411.801.841.731.572.202.623.04
Collision at 3 s (%)2.422.152.942.882.662.493.083.753.82
Avg. collision (%)1.641.461.891.891.791.682.392.842.96

As shown in Table R5, image/video self-supervised representations remain strongest overall on Bench2Drive. DINOv3 and V-JEPA 2.1 tie for the lowest average L2 error, while V-JEPA 2.1 achieves the lowest average collision rate. Foundation representations also outperform reconstruction and BEV alternatives on both average metrics. These results indicate that the main representation trends hold beyond nuPlan/NAVSIM.

We have reported the zero-shot representation comparison on Bench2Drive in Appendix B.5 of the revised paper.

Reviewer_frjbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReWAM evaluates nine visual representations within a shared pipeline for inverse dynamics, world modeling, and action planning. It also investigates world–action interaction mechanisms, proposing ETI-cache to route world-prediction gradients into the action expert during training while skipping future-state generation at inference.

Critical strengths
  1. Benchmarking nine visual representations within a unified planning pipeline across NAVSIM v1, v2, and navhard offers practical empirical guidance, complemented by decoded-video FID/FVD results in Appendix B.1 (Table 12).

  2. ETI-cache achieves a practical latency–accuracy trade-off: Table 4 reports 59 ms end-to-end latency (including the visual encoder) versus 214 ms for IWE, reaching 91.6 PDMS / 91.7 EPDMS on navtest and 37.8 versus 39.3 on navhard. The stop-gradient ablation in Appendix B.3 (Table 14) shares the exact forward pass and isolates the benefit of routing world-loss gradients into the action expert.

Critical weaknesses
  1. The proposed world-model (WM) metric is heavily confounded by the representation-specific inverse dynamics model (IDM) used to evaluate it. In Tables 1–2, WM errors closely mirror IDM errors across representations (e.g., 0.761 vs. 0.752 for DINOv3; 1.194 vs. 1.151 for BEV). Even if the world model predicted ground-truth future latents perfectly, the resulting score would still be bounded by each IDM's baseline action-recovery error. Consequently, the tight linear correlation in Figure 5 largely reflects shared evaluator error rather than an independent link between world modeling and inverse dynamics capability.

  2. High-level claims about representation families are entangled with major differences in backbone capacity, resolution, and token budgets. In Table 9, vision foundation models use ViT-L features at 17×3017\times 30 tokens, DC-AE compresses to 9×159\times 15 latents, and BEVFormer uses a ResNet-50 across eight views. A shared downstream head does not control for these upstream capacity and token-count disparities. Furthermore, the ETI-cache mechanism ablations in Table 14 (+0.6 navtest PDMS, +1.5 navhard) are reported without multi-seed variance.

Questions for authors
  1. Have you evaluated the IDM evaluator on ground-truth future latents (or under controlled temporal/action perturbations) to establish an oracle upper bound for each representation, so readers can see how much of the WM error in Tables 1–2 comes from world prediction versus IDM ceiling?

  2. Which representation takeaways hold when spatial token counts and backbone sizes are matched? Additionally, how many training seeds were used for Tables 4 and 14, and what is the seed-to-seed variance relative to the 0.6 PDMS / 1.5 navhard gains?

3 comments
AuthorsOpenReview ↗

Response to Reviewer frjb (Part 1/3)

We thank the reviewer for the constructive feedback. Below, we present additional experiments and clarify our evaluation to address the concerns.

Q1 & W1: WM evaluation confounded by a representation-specific IDM.

The proposed world-model (WM) metric is heavily confounded by the representation-specific inverse dynamics model (IDM) used to evaluate it. Consequently, the tight linear correlation in Figure 5 largely reflects shared evaluator error rather than an independent link between world modeling and inverse dynamics capability.

Have you evaluated the IDM evaluator on ground-truth future latents (or under controlled temporal/action perturbations) to establish an oracle upper bound for each representation, so readers can see how much of the WM error in Tables 1–2 comes from world prediction versus IDM ceiling?

Thank you for highlighting this concern. Table 1 in our initial manuscript already reports each representation-specific IDM evaluated on ground-truth future latents, providing an oracle-input reference for action recovery. Figure 5(a) further shows that WM errors closely track IDM errors, indicating that the WM evaluation is confounded by differences in action-recovery ability among the representation-specific IDMs. To remove this source of confounding, we therefore redesign the WM evaluation around a shared IDM action estimator.

Specifically, to use the same IDM across representations, we first decode their predicted world latents into images and re-encode them with one frozen DINOv3 encoder. This maps all predictions into a common feature space, where one frozen DINOv3-based IDM recovers the actions. We apply the same decode/re-encode path to the DINOv3 variant as well. BEV is excluded because its spatial features are not aligned with image pixels and cannot use this image-based conversion.

To test whether representation differences in reconstruction quality along the decode/re-encode path affect the shared IDM, we also evaluate a reconstruction reference for each representation. We encode and decode real future images using that representation, then apply the same shared IDM estimator. Table R1 compares the resulting reference errors with errors on generated futures, both measured against the corresponding conditioning actions.

Table R1. WM evaluation with a shared DINOv3-based IDM. Overall action error (m) ↓, following the definition in manuscript Table 2. Reconstruction references and generated futures use the same current image and evaluator.

RepresentationReconstruction-reference error ↓Generated-future error ↓
DINOv30.7680.774
V-JEPA 2.10.7730.781
SigLIP 20.7790.806
CLIP0.7810.814
DA30.7700.803
MoGe-20.7610.800
SD-VAE0.7841.365
DCAE0.7981.466

As shown in Table R1, the similar reconstruction-reference errors suggest that the DINOv3-based IDM is relatively insensitive to reconstruction differences along the decode/re-encode path. Thus it is suitable as a shared action estimator across representations. Under the shared evaluation protocol, image/video self-supervised representations achieve the lowest errors, and foundation representations substantially outperform SD-VAE and DCAE. These trends agree with our original results and show that the representation advantages are not solely driven by differences among their individual IDMs.

We have added the shared-IDM evaluation and reconstruction-reference results to Appendix B.6 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer frjb (Part 2/3)

Q2 & W2-1: Representation differences in backbone sizes and token counts.

The shared downstream architecture does not control for upstream differences: foundation encoders use ViT-L features on a 17 × 30 grid, DCAE produces 9 × 15 latents, and BEVFormer uses a ResNet-50 with eight camera views. Identify which representation takeaways still hold when backbone sizes and spatial token counts are matched.

We appreciate the suggestion to better control these factors. We add a more controlled comparison across IDM, WM, and WAM to reduce backbone-size differences and align spatial token counts.

  • Backbone size: We replace the approximately 300M-parameter ViT-L foundation encoders with ViT-B models of approximately 86M parameters. SD-VAE (34M), DCAE (123M), and BEVFormer (48M) retain their original encoders. This places the encoders in a comparable size range, with no representation family consistently having more parameters than the others.
  • Spatial token count: All non-BEV representations use a 9 × 15 grid (135 tokens per frame). We apply spatial adaptive average pooling to the 17 × 30 foundation features and grouped SD-VAE latents, while DCAE already has the target grid. BEVFormer retains its native 32 × 40 grid (1,280 tokens) because the spatial definition of ground-plane BEV tokens inherently differs from the image-plane grids used by the other representations.

Tables R2–R4 report the results, and Table R5 summarizes the encoder sizes and spatial tokenization.

Table R2. IDM action recovery under the adjusted settings (corresponding to manuscript Table 1).

MetricDINOv3V-JEPA 2.1SigLIP 2CLIPDA3MoGe-2SD-VAEDCAEBEVFormer
ADE (m) ↓0.5720.5940.7100.7530.7200.6130.9550.9770.919
FDE (m) ↓0.9410.9361.1891.2040.9831.0301.2271.2701.383
Overall (m) ↓0.7570.7650.9500.9790.8520.8221.0911.1241.151

Table R3. Appearance-agnostic WM evaluation via world-action consistency under the adjusted settings (corresponding to manuscript Table 2).

MetricDINOv3V-JEPA 2.1SigLIP 2CLIPDA3MoGe-2SD-VAEDCAEBEVFormer
ADE (m) ↓0.5880.6740.7260.7790.7280.6410.9801.0010.942
FDE (m) ↓0.9390.9271.2131.2400.9901.0591.2621.3191.446
Overall (m) ↓0.7640.8010.9701.0100.8590.8501.1211.1601.194

Table R4. WAM planning under the adjusted settings (corresponding to manuscript Table 3).

MetricDINOv3V-JEPA 2.1SigLIP 2CLIPDA3MoGe-2SD-VAEDCAEBEVFormer
PDMS (v1) ↑91.591.390.089.690.490.288.688.489.0
EPDMS (v2) ↑91.491.490.389.991.090.889.188.988.8
EPDMS (navhard) ↑38.737.433.233.735.835.131.030.729.1

Table R5. Encoder sizes and spatial tokenization in the adjusted comparison.

RepresentationBackbone / encoder sizeSpatial tokenization per frame
DINOv3ViT-B/16, ~86MAdaptive pooling to 9 × 15 (135)
V-JEPA 2.1ViT-B/16, ~86MAdaptive pooling to 9 × 15 (135)
SigLIP 2ViT-B/16, ~86MAdaptive pooling to 9 × 15 (135)
CLIPViT-B/16, ~86MAdaptive pooling to 9 × 15 (135)
DA3ViT-B/14, ~86MAdaptive pooling to 9 × 15 (135)
MoGe-2ViT-B/14, ~86MAdaptive pooling to 9 × 15 (135)
SD-VAEKL-f8, ~34MAdaptive pooling to 9 × 15 (135)
DCAEf32-c32, ~123MNative 9 × 15 (135)
BEVFormerR50-base, ~48MNative 32 × 40 (1,280)

Tables R2–R4 show modest performance changes after these adjustments, while the original trends across representation families remain consistent. Image/video self-supervised representations remain strongest overall. Foundation representations retain a substantial advantage over SD-VAE, DCAE, and BEV despite the reductions in backbone capacity and spatial token budget. Even with a larger token budget, BEVFormer remains far behind foundation representations, highlighting limitations of this driving-specific representation for world-action modeling. These results suggest that their advantage is not simply due to larger encoders or more tokens, supporting the importance of representation pretraining objectives for world-action modeling.

We have included the controlled comparison and adjusted encoder configurations in Appendix B.7 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer frjb (Part 3/3)

Q2 & W2-2: Multi-seed variance of ETI-cache training.

State the number of training seeds used for Tables 4 and 14 and report seed-to-seed variance. In particular, establish whether the reported 0.6 PDMS and 1.5 navhard EPDMS gains in the stop-gradient comparison are robust relative to training variability.

Thank you for suggesting this robustness check. The original results in manuscript Tables 4 and 14 used one training seed. Following your suggestion, we repeat training with three random seeds for the interaction variants and the ETI-cache stop-gradient ablation, using the original manuscript configurations and evaluation protocol. Table R6 reports mean ± std and 95% t-based confidence intervals.

Table R6. Training-seed variability of the comparisons in manuscript Tables 4 and 14. All variants use DINOv3. Each cell gives mean ± std followed by [95% t-based CI].

VariantPDMS (v1) ↑EPDMS (v2) ↑EPDMS (navhard) ↑
IWE91.5928 ± 0.0119 [91.5633, 91.6223]91.6995 ± 0.0291 [91.6273, 91.7717]39.3474 ± 0.0117 [39.3184, 39.3763]
ITE91.4757 ± 0.0126 [91.4445, 91.5070]91.5431 ± 0.0311 [91.4659, 91.6203]38.5038 ± 0.0160 [38.4641, 38.5435]
EO88.3762 ± 0.0263 [88.3108, 88.4415]88.7227 ± 0.0318 [88.6437, 88.8017]30.5558 ± 0.0217 [30.5018, 30.6098]
DIE90.9499 ± 0.0217 [90.8961, 91.0037]91.2155 ± 0.0336 [91.1320, 91.2991]35.9837 ± 0.0237 [35.9249, 36.0425]
ETI-sync91.2182 ± 0.0240 [91.1587, 91.2777]91.2848 ± 0.0271 [91.2174, 91.3522]36.4038 ± 0.0249 [36.3419, 36.4657]
ETI-cache91.6262 ± 0.0273 [91.5585, 91.6940]91.7205 ± 0.0232 [91.6630, 91.7780]37.9876 ± 0.0136 [37.9538, 38.0213]
ETI-cache (stop-grad)90.9488 ± 0.0325 [90.8680, 91.0296]91.1790 ± 0.0191 [91.1316, 91.2265]36.2867 ± 0.0132 [36.2540, 36.3194]

As shown in Table R6, ETI-cache consistently outperforms its stop-gradient variant on all three planning benchmarks. The standard deviations across seeds are much smaller than the reported 0.6 PDMS and 1.5 navhard EPDMS gains, supporting the stability of these improvements. Its advantages over DIE and ETI-sync also remain consistent across seeds.

We have reported the multi-seed interaction and stop-gradient results in Appendix B.8 of the revised paper.

Reviewer_PHaCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReWAM, a unified framework for analyzing the influence of world representations on autonomous driving planning. It systematically evaluates nine representations across inverse dynamics modeling (IDM), world modeling (WM), and world-action modeling (WAM). The authors also introduce an appearance-agnostic world-action consistency metric and an execute-then-imagine (ETI) strategy that transfers world-prediction supervision to action generation while enabling action-only inference. Experiments on NAVSIM demonstrate strong planning performance, achieving 91.6 PDMS on navtest and 39.3 EPDMS on navhard. The efficient ETI-cache variant achieves 59 ms inference latency while maintaining competitive planning accuracy.

Critical strengths
  1. The unified framework enables comparisons across nine world representations and three related modeling tasks, providing useful insights into representation selection for planning.

  2. The proposed ETI-cache mechanism allows world-prediction supervision to improve action generation without requiring future-world generation during inference.

  3. ReWAM achieves competitive results on NAVSIM benchmarks, with substantial inference efficiency improvements over joint world-action generation.

  4. The paper includes representation comparisons, interaction ablations, zero-shot transfer experiments, and qualitative long-horizon world generation results.

Critical weaknesses

1.The framework primarily integrates existing pretrained representations, MoT-based diffusion, and RAE-style architectures. The ETI mechanism is promising, but its conceptual distinction from existing auxiliary world-model supervision and action-conditioned prediction methods requires stronger justification.

2.Different encoders have different pretraining objectives, input modalities, temporal contexts, and representation structures. Consequently, the observed performance differences cannot be attributed solely to representation quality.

3.World-action consistency is measured using a representation-specific frozen IDM. Since IDM accuracy and WM consistency are evaluated through closely related models, their extremely high correlation (r = 0.999) may partly reflect shared representation-dependent errors rather than independent evidence of world-model quality.

  1. The paper reports strong benchmark improvements but provides insufficient analysis of training-seed variance, confidence intervals, and statistical significance. This is particularly important for the relatively small differences among the strongest representations and interaction variants.

  2. Zero-shot transfer is evaluated using trajectory errors and collision metrics, while long-horizon world generation is primarily assessed qualitatively. More rigorous closed-loop evaluations and quantitative long-horizon consistency measurements are needed.

Questions for authors

1.How can the authors disentangle the effects of representation quality from differences in encoder pretraining, temporal context, input modalities, and tokenization?

2.Can the authors validate world-action consistency using an independent action estimator or alternative metric to rule out representation-specific IDM bias?

3.What is the fundamental novelty of ETI-cache compared with existing action-conditioned world prediction and auxiliary world-model supervision methods?

4.Can the authors report multi-seed results and statistical significance for the representation comparisons and ETI ablations?

5.How does ReWAM perform in closed-loop driving scenarios, and can the authors provide quantitative evaluations of long-horizon physical consistency and error accumulation?

10 comments
AuthorsOpenReview ↗

Response to Reviewer PHaC (Part 1/4)

We thank the reviewer for the constructive feedback and address each concern below.

Q1 & W2: Confounding factors in representation comparisons.

Encoders differ in pretraining objectives, input modalities, temporal context, representation structure, and tokenization. Explain how these factors are disentangled from representation quality when interpreting the observed performance differences.

Thank you for raising this point. We add a controlled comparison across IDM, WM, and WAM to reduce differences in encoder size, temporal context, tokenization, and input modality while retaining different pretraining objectives.

The original foundation encoders use ViT-L (~300M), versus 34M for SD-VAE, 123M for DCAE, and 48M for BEVFormer. V-JEPA 2.1 and DA3 use multiple frames. All representations use 510 tokens per frame, except DCAE (135) and BEVFormer (1,280). Only BEVFormer uses eight cameras.

For a more controlled comparison, we make the following changes:

  • Encoder size: Foundation encoders use ViT-B (~86M). SD-VAE, DCAE, and BEVFormer are unchanged, removing a consistent encoder capacity advantage for foundation encoders.
  • Temporal context: All encoders process one frame independently, with temporal patch size 1 for V-JEPA 2.1 and framewise DA3 encoding. Downstream history and prediction horizons remain aligned.
  • Tokenization: All non-BEV representations use a 9 × 15 grid (135 tokens per frame). Spatial adaptive average pooling maps the 17 × 30 foundation features and grouped SD-VAE latents to this grid. DCAE already has this grid.
  • Input modality: Non-BEV encoders use one front camera. BEVFormer retains eight cameras and its 32 × 40 grid because ground-plane BEV tokens differ spatially from image-plane patches.

Table R1. IDM action recovery under the adjusted settings (corresponding to manuscript Table 1).

MetricDINOv3V-JEPA 2.1SigLIP 2CLIPDA3MoGe-2SD-VAEDCAEBEVFormer
ADE (m) ↓0.5720.5940.7100.7530.7200.6130.9550.9770.919
FDE (m) ↓0.9410.9361.1891.2040.9831.0301.2271.2701.383
Overall (m) ↓0.7570.7650.9500.9790.8520.8221.0911.1241.151

Table R2. Appearance-agnostic WM evaluation via world-action consistency under the adjusted settings (corresponding to manuscript Table 2).

MetricDINOv3V-JEPA 2.1SigLIP 2CLIPDA3MoGe-2SD-VAEDCAEBEVFormer
ADE (m) ↓0.5880.6740.7260.7790.7280.6410.9801.0010.942
FDE (m) ↓0.9390.9271.2131.2400.9901.0591.2621.3191.446
Overall (m) ↓0.7640.8010.9701.0100.8590.8501.1211.1601.194

Table R3. WAM planning under the adjusted settings (corresponding to manuscript Table 3).

MetricDINOv3V-JEPA 2.1SigLIP 2CLIPDA3MoGe-2SD-VAEDCAEBEVFormer
PDMS (v1) ↑91.591.390.089.690.490.288.688.489.0
EPDMS (v2) ↑91.491.490.389.991.090.889.188.988.8
EPDMS (navhard) ↑38.737.433.233.735.835.131.030.729.1

Table R4. Adjusted encoder configurations.

RepresentationPretraining objectiveEncoder sizeTemporal contextTokenization per frameInput modality
DINOv3Image self-supervisionViT-B/16, ~86M1 frameAdaptive pooling to 9 × 15 (135)1 front camera
V-JEPA 2.1Video predictive self-supervisionViT-B/16, ~86M1 frameAdaptive pooling to 9 × 15 (135)1 front camera
SigLIP 2Language alignment supervisionViT-B/16, ~86M1 frameAdaptive pooling to 9 × 15 (135)1 front camera
CLIPLanguage alignment supervisionViT-B/16, ~86M1 frameAdaptive pooling to 9 × 15 (135)1 front camera
DA3Geometry supervisionViT-B/14, ~86M1 frameAdaptive pooling to 9 × 15 (135)1 front camera
MoGe-2Geometry supervisionViT-B/14, ~86M1 frameAdaptive pooling to 9 × 15 (135)1 front camera
SD-VAEReconstruction supervisionKL-f8, ~34M1 frameAdaptive pooling to 9 × 15 (135)1 front camera
DCAEReconstruction supervisionf32-c32, ~123M1 frameNative 9 × 15 (135)1 front camera
BEVFormerDriving perceptionR50-base, ~48M1 frameNative 32 × 40 (1,280)8 cameras

Tables R1–R3 show modest changes after alignment, but the representation-family trends remain consistent. Image/video self-supervised representations remain strongest overall. Foundation representations still substantially outperform SD-VAE, DCAE, and BEV despite reduced capacity and token budgets. These results suggest that their advantage is not simply due to larger encoders, more tokens, or temporal context, supporting the importance of representation pretraining objectives for world-action modeling.

We have added this controlled comparison and the encoder configurations to Appendix B.7 of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer PHaC (Part 2/4)

Q2 & W3: Representation-specific IDM bias in WM evaluation.

World-action consistency is evaluated through a frozen IDM specific to each representation. The very high IDM–WM correlation may partly reflect shared representation-dependent errors rather than independent evidence of world-model quality. Validate the metric using an independent action estimator or an alternative evaluation measure.

Thank you for the suggestion. We add a new WM assessment with a shared IDM as the action estimator to remove bias in action-recovery ability among the representation-specific IDMs.

To use one IDM across representations, we decode predicted latents into images and re-encode them with a frozen DINOv3 encoder. One frozen DINOv3-based IDM then recovers actions in this shared space. DINOv3 follows the same decode/re-encode path. BEV is excluded because its features are not pixel-aligned and have no RGB decoder.

To test whether reconstruction differences among representations affect this readout, we encode and decode real future images with each representation and apply the same shared estimator. Table R5 compares these reconstruction-reference errors with generated-future errors against the conditioning actions.

Table R5. WM evaluation with a shared DINOv3-based IDM. Overall action error (m) ↓, following the definition in manuscript Table 2. Reconstruction references and generated futures use the same current image and evaluator.

RepresentationReconstruction-reference error ↓Generated-future error ↓
DINOv30.7680.774
V-JEPA 2.10.7730.781
SigLIP 20.7790.806
CLIP0.7810.814
DA30.7700.803
MoGe-20.7610.800
SD-VAE0.7841.365
DCAE0.7981.466

Table R5 shows similar reference errors, suggesting that the DINOv3-based IDM is relatively insensitive to reconstruction differences and suitable as a shared estimator. Image/video self-supervised representations achieve the lowest errors, and foundation representations substantially outperform SD-VAE and DCAE. These trends agree with the original results and are not solely driven by differences among the individual IDMs.

We have included the shared-IDM evaluation and reconstruction-reference results in Appendix B.6 of the revised paper.

Q3 & W1: Positioning vs. prior work and novelty.

The framework combines existing pretrained representations, MoT-based diffusion, and RAE-style architectures. Clarify its contribution beyond integrating these components and explain the conceptual distinction between ETI-cache and existing action-conditioned world prediction or auxiliary world-model supervision methods.

Thank you for raising this point. Our central contribution is a unified world-action modeling framework that supports multiple representations, tasks, and interaction paradigms, enabling controlled and fair comparisons of key WAM design choices. The architecture of RAE inspired our support for diverse representations, but was originally designed for image generation. We extend it with a MoT-DDT architecture for joint world-action modeling, task-specific attention masks, and configurable world-action interactions to support planning-oriented tasks within one framework.

ETI-cache addresses a problem identified through this framework: how to retain the planning benefits of world modeling without generating future worlds at inference. Rather than adopting one interaction design, we systematically compare alternatives and identify a direct gradient path from world prediction to the action expert as a key factor in balancing planning performance and efficiency. Compared with simulation-oriented action-conditioned world prediction, ETI-cache also predicts the actions, with the goal of improving planning. Compared with auxiliary world-model pretraining, such as DriveVLA-W0, which uses dense world supervision during pretraining but removes it in the action fine-tuning stage, ETI-cache retains joint world-action training. The resulting model supports both planning and driving simulation.

Our experiments further show why the specific ETI-cache design is non-trivial. Simply changing the attention mask, as in ETI-sync, does not close the planning performance gap. ETI-cache instead differentiably rolls out clean actions and uses their layerwise K/V cache to condition future-world prediction. This provides a path for world-prediction gradients to reach the action expert through its own predictions. The sync/cache comparison and stop-gradient ablation support the importance of these choices beyond the general idea of adding world-model supervision.

Thus, ETI-cache is not only a novel interaction method but also an outcome of systematic analysis of world-action interaction paradigms within our unified framework.

AuthorsOpenReview ↗

Response to Reviewer PHaC (Part 3/4)

Q4 & W4: Training variance and statistical reliability.

Report multi-seed results, confidence intervals, and statistical significance for the representation comparisons and ETI ablations, particularly where differences among the strongest representations or interaction variants are small.

We appreciate the suggestion to assess training variability. We repeat training with three random seeds for the two strongest representations, DINOv3 and V-JEPA 2.1, and the interaction variants below. All runs follow the same training and evaluation protocols. Each cell in Tables R6–R7 gives mean ± std followed by [95% t-based CI].

Table R6. Training-seed variability of the two strongest representations.

RepresentationIDM Overall (m) ↓WM consistency Overall (m) ↓PDMS (v1) ↑EPDMS (v2) ↑EPDMS (navhard) ↑
DINOv30.75144 ± 0.00221 [0.74595, 0.75692]0.76407 ± 0.00138 [0.76063, 0.76751]91.5928 ± 0.0119 [91.5633, 91.6223]91.6995 ± 0.0291 [91.6273, 91.7717]39.3474 ± 0.0117 [39.3184, 39.3763]
V-JEPA 2.10.77018 ± 0.00197 [0.76530, 0.77506]0.79737 ± 0.00493 [0.78512, 0.80962]91.2464 ± 0.0328 [91.1649, 91.3280]91.4930 ± 0.0217 [91.4392, 91.5469]36.8786 ± 0.0184 [36.8329, 36.9243]

Table R7. Training-seed variability of interaction variants using DINOv3.

VariantPDMS (v1) ↑EPDMS (v2) ↑EPDMS (navhard) ↑
IWE91.5928 ± 0.0119 [91.5633, 91.6223]91.6995 ± 0.0291 [91.6273, 91.7717]39.3474 ± 0.0117 [39.3184, 39.3763]
ITE91.4757 ± 0.0126 [91.4445, 91.5070]91.5431 ± 0.0311 [91.4659, 91.6203]38.5038 ± 0.0160 [38.4641, 38.5435]
EO88.3762 ± 0.0263 [88.3108, 88.4415]88.7227 ± 0.0318 [88.6437, 88.8017]30.5558 ± 0.0217 [30.5018, 30.6098]
DIE90.9499 ± 0.0217 [90.8961, 91.0037]91.2155 ± 0.0336 [91.1320, 91.2991]35.9837 ± 0.0237 [35.9249, 36.0425]
ETI-sync91.2182 ± 0.0240 [91.1587, 91.2777]91.2848 ± 0.0271 [91.2174, 91.3522]36.4038 ± 0.0249 [36.3419, 36.4657]
ETI-cache91.6262 ± 0.0273 [91.5585, 91.6940]91.7205 ± 0.0232 [91.6630, 91.7780]37.9876 ± 0.0136 [37.9538, 38.0213]
ETI-cache (stop-grad)90.9488 ± 0.0325 [90.8680, 91.0296]91.1790 ± 0.0191 [91.1316, 91.2265]36.2867 ± 0.0132 [36.2540, 36.3194]

Tables R6–R7 show small standard deviations and narrow confidence intervals. DINOv3 remains ahead of V-JEPA 2.1 across tasks. ETI-cache consistently outperforms DIE, ETI-sync, and its stop-gradient variant on all three benchmarks, with gaps larger than seed variation. The non-overlapping 95% CIs further support the statistical significance of these differences.

We have reported these multi-seed results and confidence intervals in Appendix B.8 of the revised paper.

Q5 & W5: Closed-loop validation and quantitative long-horizon evaluation.

Zero-shot transfer is assessed through trajectory errors and collision metrics, while long-horizon generation is evaluated mainly qualitatively. Provide closed-loop driving evaluation and quantitative measurements of long-horizon physical consistency and error accumulation.

Thank you for recommending these additional evaluations.

(1) Zero-shot closed-loop validation. We add a zero-shot comparison with DriveVLA-W0, PWM, and DriveVA on HUGSIM, a photorealistic closed-loop simulation benchmark built from reconstructed real-world driving scenes. We use the NAVSIM-trained models without target-domain fine-tuning. All methods use the same scenarios, controller, traffic seeds, and scoring protocol. The simulator executes each predicted plan and returns updated observations for the next planning step. Table R8 reports NC, DAC, TTC, comfort, PDMS, and Route Completion (RC) over all 436 scenarios, together with HUGSIM Driving Score (HD-Score) by difficulty and overall.

Table R8. Zero-shot closed-loop driving on HUGSIM. NC, DAC, TTC, comfort, PDMS, and RC summarize all 436 scenarios. HD-Score is reported by difficulty and overall. All scores use a 0–100 scale, and higher is better. Domain counts: nuScenes (88), Waymo (108), KITTI-360 (113), and PandaSet (127).

MethodNC ↑DAC ↑TTC ↑Comfort ↑PDMS ↑RC ↑HD-Score Easy ↑HD-Score Medium ↑HD-Score Hard ↑HD-Score Extreme ↑HD-Score Avg. ↑
DriveVLA-W042.1592.6332.5178.6340.3640.9452.6536.2224.4312.8231.11
PWM42.3490.8931.5272.2135.6339.8655.2634.7422.1910.4530.00
DriveVA54.6492.3752.4877.6251.2550.9174.0446.2023.7912.6338.44
ReWAM66.3195.1459.1175.4655.4055.0777.2151.6529.4413.5842.46

As shown in Table R8, ReWAM achieves the highest PDMS, RC, and overall HD-Score among the compared methods, and leads in HD-Score at every difficulty level. These results extend our open-loop zero-shot findings to closed-loop driving.

We have added this closed-loop evaluation to Appendix C.2 of the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for Area Chair

Dear Area Chair,

Thank you and the reviewers for your time and constructive feedback. We summarize the recognized strengths, our responses, and the discussion outcomes below.

Strengths recognized across the reviewers

  • A unified comparison of nine world representations across inverse dynamics, world modeling, and planning provides practical guidance for representation selection (frjb, dwyQ, PHaC).
  • ETI-cache combines world-prediction supervision during training with action-only inference, delivering strong planning performance at low latency (frjb, dwyQ, PHaC).
  • ReWAM achieves competitive NAVSIM results (PHaC) and strong planning with a single camera and a much smaller model than other world-action models (dwyQ).
  • ReWAM shows strong zero-shot transfer to Bench2Drive and generates stable minute-long rollouts in the full feature space of DINOv3 (dwyQ).

Reviewer frjb (rating: 3, confidence: 3)

  • Strengths: ReWAM's nine-representation benchmark offers "practical empirical guidance". ETI-cache achieves "a practical latency–accuracy trade-off" at 59 ms, and the stop-gradient ablation isolates "the benefit of routing world-loss gradients into the action expert".
  • Concerns: Representation-specific IDM confounding, differences in encoder capacity and token budgets, and training-seed variability of the ETI-cache gains.
  • Our Response: Added a shared-IDM evaluation with reconstruction references and controlled encoder/token comparisons. Both retain the original representation trends. We also reported training-seed variability and confidence intervals for the interaction variants and stop-gradient ablation.
  • Reviewer's Feedback: No follow-up reply to our rebuttal at the time of this summary.

Reviewer dwyQ (rating: 3, confidence: 4)

  • Strengths: ReWAM provides a "controlled and fair comparison of nine world representations", achieves "strong planning with a single camera and a much smaller model", and "transfers zero-shot to Bench2Drive" with lower trajectory error and collision rates.
  • Concerns: Location recall, representation-specific IDM bias, representation rankings at unseen locations, quantitative long-horizon evidence and action control, and whether direct trajectory supervision could replace world prediction in ETI-cache.
  • Our Response: Addressed these concerns by clarifying data and rollout protocols, retraining under a city-disjoint split, and adding shared-IDM evaluation, zero-shot representation rankings, and an ablation with direct trajectory supervision and no world prediction. We further evaluated minute-long rollouts on 141 held-out Boston sequences, reran the original showcase scenes with the city-disjoint model, and provided different-action rollouts from the same history at unseen locations. The representation trends and long-horizon advantages persist without location overlap with our training recordings.
  • Reviewer's Feedback: After these additions, dwyQ wrote "I am satisfied with the results", noted that generalization to unseen locations was maintained, and highlighted: "To my knowledge, this is also the first work to show stable minute-long driving rollouts in the full feature space of a pretrained foundation model like DINOv3". The reviewer concluded "I will raise my score" and increased the rating from 2 to 3, with high confidence (4). We will emphasize this capability more explicitly in the contribution statement, as suggested.

Reviewer PHaC (rating: 3, confidence: 3)

  • Strengths: ReWAM provides "useful insights into representation selection for planning" and achieves "competitive results on NAVSIM benchmarks" with "substantial inference efficiency improvements". ETI-cache improves action generation without generating future worlds at inference.
  • Concerns: Novelty relative to prior work, confounding factors in representation comparisons, IDM bias, statistical reliability, and the need for closed-loop and quantitative long-horizon validation.
  • Our Response: Addressed all five concerns through controlled representation comparisons, shared-IDM evaluation with reconstruction references, clarification of the framework's contributions and ETI-cache's distinction from prior approaches, and reported seed variability with confidence intervals. We also added zero-shot closed-loop evaluation on HUGSIM and quantitative measurements of long-horizon video quality, physical consistency, and error accumulation. The representation trends persist, ReWAM leads in overall HUGSIM driving score, and DINOv3 shows less degradation over long rollouts.
  • Reviewer's Feedback: Following our responses, PHaC replied and raised the rating: "Thank you for your reply. I have no further questions."

We hope this summary helps your assessment of the paper in light of the additional evidence and the reviewers' feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.