acceptodds
Under review as a conference paper at ICLR 2027

SimWAM: A Simple World Action Model for End-to-End Autonomous Driving

Abstract

In autonomous driving, World-Action Models (WAMs) have improved end-to-end planning by transferring video dynamics priors to action prediction, but many still couple planning with future-video generation at inference, incurring substantial computational overhead. We present SimWAM, a simple yet effective WAM that leverages future-video prediction solely as a training-time supervision signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without future frame generation at inference. This design supports multiple pretrained video backbones and independent action-expert scaling within a shared attention interface, while preserving the joint learning objective. Moreover, we apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Experiments show that SimWAM achieves PDMS on NAVSIM with a favorable trade-off between accuracy and latency among world-model-based planners, while transferring zero-shot to nuScenes. It also achieves competitive planning accuracy on WOD-E2E and PhysicalAI-Autonomous-Vehicles. These results position SimWAM as a plain yet solid baseline for efficient autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, End-to-End Autonomous Driving, Planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4NDhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies end-to-end driving with a world-action model that co-trains a pretrained video DiT and a lightweight action DiT by joint flow matching. An isolated attention mask lets future-video prediction supervise the current-frame representation during training, while inference predicts a trajectory from the current image, ego state, and navigation command without generating future frames. Reinforcement learning then tunes the action expert on a driving reward. The central claim is that this training-time world model is enough for strong, low-latency planning. The evidence is NAVSIM Table 1 and Tables 6–9, the NAVSIM-v2 results in Tables 10–11, and the transfers in Tables 3–5.

Critical strengths
  1. The inference interface matches the claim. Equation 3 is the imagine-then-act factorization used by prior driving WAMs; Equation 4 removes future latents from the policy. Figure 2 and Section 4.1 keep future-frame tokens and action tokens mutually invisible, so the action expert reads only the current-frame representation. Table 6 shows the consequence: the isolated mask reaches 90.3 PDMS at 518 ms, against 90.2 PDMS at 1512 ms for bidirectional attention and 90.1 PDMS at 1499 ms for action-to-video attention.
  2. Future-video supervision has a measured effect on the planner, not only on video quality. In Table 9, fine-tuning the video expert with the action loss moves PDMS from 86.5 to 88.4, and adding future-video supervision moves it to 90.3. Figure 3 probes the same current-frame features: the co-trained representation lowers ADE for both surrounding-vehicle motion and the ego trajectory relative to the two ablations without that supervision.
  3. The decoupled interface is tested rather than only described. Table 7 swaps in LTX-Video, Wan2.1-1.3B, Cosmos-Predict2.5, and Wan2.2-5B without changing the action expert. Cosmos-Predict2.5, the driving-pretrained backbone, reaches 90.4 PDMS under imitation and 92.4 after reinforcement learning, above Wan2.2-5B at 91.9. Table 8 scales the action expert from 0.21B to 1.02B and moves imitation PDMS from 89.9 to 90.3.
  4. The planning result is not a single NAVSIM number. Table 1 reports 91.9 PDMS with one front camera, 0.8 above SGDrive and 1.0 above DriveVA. On the same 65K-sample, 1,000-clip PhysicalAI-AV protocol, Table 4 reports lower ADE and FDE than DriveWAM without test-time future-frame synthesis. Appendix Table 11 reports 37.6 EPDMS on reactive navhard, against 30.6 for DriveLaW. Figure 4 and Table 15 tie the reinforcement-learning gain to late-step SDE noise and the imitation anchor: removing the anchor drops PDMS to 84.3.
Critical weaknesses
  1. The paper frames SimWAM as a simple baseline rather than a new world-model algorithm, and the headline number should be read that way. The training-time split between video supervision and test-time imagination is the idea credited to Fast-WAM. Table 1’s 91.9 PDMS is the reinforcement-learning model in Table 9 (ID 4); the imitation model is 90.3. Table 1 does not mark the reinforcement-learning stage, and the system uses a Wan2.2-5B video expert plus a 1.02B action expert. The 0.8 PDMS margin over SGDrive is real on the reported split, but it is not a large algorithmic gap.
  2. The future-video contribution is isolated only before reinforcement learning. Table 9’s +1.9 PDMS (88.4 to 90.3) has no paired run that applies the same reward optimization to the action-only video fine-tune. The abstract’s 91.9 therefore mixes the world-model signal with the reward stage. Table 8’s scaling gain under imitation is 0.4 PDMS for a fourfold increase in action-expert FLOPs, so capacity alone does not explain Table 9, but the post-RL attribution remains incomplete.
  3. Two reported gains are too small, or too weakly held out, to carry the paper. In Table 6 the isolated mask wins by 0.1 PDMS; the supported result is the latency reduction. On WOD-E2E, reinforcement learning is run on the validation split, so the validation RFS of 8.29 is not a held-out score. On the test split the same stage moves RFS from 7.77 to 7.84 and ADE@5s from 2.69 to 2.68. No seed variance is reported for the 0.8 PDMS margin on NAVSIM.
Questions for authors

Were the Table 1 world-model and VLM rows re-evaluated in this codebase on navtest with the stated camera layouts, and does the 0.8 PDMS margin over SGDrive hold across a second seed? Some suggestions:

  1. Mark Table 1 as the reinforcement-learning model and report the 90.3 imitation PDMS beside it.
  2. Write the scalar reward used for ri in Equation 7, including which PDMS terms enter it.
  3. State in the Figure 1 caption whether the plotted latency is the 10-step imitation setting from Table 6.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4NDh 1/2

We thank the reviewer for the thoughtful and constructive feedback. Our responses to each point are provided below.

To W1: Comparison with SGDrive

R: Thank you for the comments. our SimWAM is a simple yet solid baseline that proves effective across multiple driving benchmarks. We have distinguished the IL and RL stages in Table 1 and compare SimWAM with SGDrive under both IL and RL. Since SGDrive does not report RL results on NAVSIM-v2, we reproduce them. The results are as follows:

MethodStageNAVSIM-v1 navtest PDMSNAVSIM-v2 navtest EPDMSNAVSIM-v2 navhard EPDMSLatency (ms)
SGDriveIL (SFT)87.486.227.1757
SGDriveRL91.183.026.9757
SimWAM (ours)IL90.390.237.6518
SimWAM (ours)RL91.990.838.8518

On NAVSIM-v1, SimWAM outperforms SGDrive by 2.9 PDMS under IL and 0.8 PDMS after RL. More remarkably, SimWAM-IL exceeds SGDrive-IL by 4.0 EPDMS on NAVSIM-v2 navtest and substantially by 10.5 EPDMS on navhard featuring safety-critical scenarios with reactive traffic, demonstrating the superior planning performance of SimWAM. Notably, the RL of SGDrive even lowers its EPDMS on NAVSIM-v2, whereas our RL further improves it on both navtest and navhard. Additionally, SimWAM runs 32% faster than SGDrive (518 vs. 757 ms).

To W2: Future-video gain before and after RL

R: We thank the reviewer for pointing out this missing comparison. Following your suggestion, we compare the IL and RL results of the three configurations in Table 9. The results are as follows:

IDVideo Expert TrainingFuture-Video SupervisionIL+ RL
1––86.588.9
2✓–88.490.2
3 (ours)✓✓90.391.9

Across the three configurations, RL consistently improves PDMS, yet ID 2 still falls behind ID 3 after RL. Future-video supervision contributes +1.9 PDMS under IL and still +1.7 PDMS after RL (91.9 vs. 90.2). This demonstrates that future-video supervision and RL are complementary, and both contribute to the final performance of SimWAM. We have extended Table 9 with the RL results in the revised paper.

To W3: Results and seed variance

R: Thank you for these helpful comments. Regarding the mask ablation, we agree that the PDMS difference among the masks in Table 6 is marginal. More importantly, we adopt the isolated mask to avoid explicitly generating future frames at test time, thereby reducing the latency. As a result, it runs nearly 3× faster than the other masks while achieving comparable or even slightly better PDMS.

Regarding WOD-E2E, we agree that the validation RFS is not held-out, since we follow NoRD to use it as the RL reward. On the held-out test split, SimWAM achieves the highest RFS of 7.84 and the lowest ADE@5s of 2.68 among all compared methods, outperforming NoRD (7.71 RFS), which also uses the RFS for RL.

Regarding the seed variance, we re-run SimWAM with five seeds and report the mean ± std on NAVSIM-v1 navtest (PDMS), NAVSIM-v2 navtest and navhard (EPDMS).

MethodNAVSIM-v1 navtest PDMSNAVSIM-v2 navtest EPDMSNAVSIM-v2 navhard EPDMS
SGDrive-IL87.486.227.1
SGDrive-RL91.183.026.9
SimWAM-IL90.32 ± 0.0890.19 ± 0.1737.62 ± 0.12
SimWAM-RL91.90 ± 0.0590.77 ± 0.1238.81 ± 0.65

The 0.8 PDMS margin over the reported 91.1 of SGDrive holds across all five seeds, with SimWAM-RL at 91.90 ± 0.05 PDMS. Notably, SimWAM-IL also surpasses SGDrive substantially on NAVSIM-v2, by 4.0 EPDMS on navtest and 10.5 EPDMS on navhard over SGDrive-IL.

AuthorsOpenReview ↗

Response to Reviewer 4NDh 2/2

To Question: Results in Table 1 and seed variance

R: Thanks for this valuable question. Regarding the results in Table 1, we would like to clarify that they are taken from the original papers (e.g., ReCogDrive and DriveWAM) with their reported camera layouts.

Regarding the second seed, as discussed in our response to W3, the 0.8 PDMS margin over the reported 91.1 of SGDrive holds across all five seeds, with SimWAM-RL at 91.90 ± 0.05 PDMS. Notably, SimWAM-IL also surpasses SGDrive substantially on NAVSIM-v2, by 4.0 EPDMS on navtest and 10.5 EPDMS on navhard over SGDrive-IL.

To Suggestion 1: Marking IL and RL in Table 1

R: Following your suggestion, we have distinguished the IL and RL results in Table 1 and Fig. 1 of the revised manuscript.

To Suggestion 2: Scalar reward formulation

R: Thanks for pointing out this omission. The reward rir_i in Eq. 7 is the official PDM score of NAVSIM, including NC, DAC, EP, TTC and Comfort, with the EP weight raised from 5 to 10 during training. We have specified the reward in of the revised manuscript.

To Suggestion 3: Latency setting in Fig. 1

R: Thank you for this useful suggestion. We would like to clarify that the latency in Fig. 1 is measured under the 10-step IL setting of Table 6 (518 ms), and the RL model has the same latency since RL does not change the inference process.

We hope these responses address your concerns, and we would be glad to discuss any remaining questions.

I think most of my concerns are addressed. Therefore, I tend to maintain my score.

Thank you very much for your thoughtful review. We sincerely appreciate your positive assessment.

Reviewer_qab8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SimWAM, an efficient World Action Model for end-to-end autonomous driving. It jointly trains a pretrained video generation expert and a lightweight action expert using flow matching, while introducing an isolated attention mechanism that eliminates explicit future-video generation during inference. The authors further employ SDE-based GRPO to optimize trajectory planning beyond imitation learning. Extensive experiments on NAVSIM, WOD-E2E, PhysicalAI-AV, and nuScenes demonstrate competitive planning performance and improved inference efficiency.

Critical strengths
  1. The paper addresses the substantial inference overhead of existing World Action Models, offering an efficient alternative without explicit test-time future generation.
  2. SimWAM achieves competitive performance across multiple driving benchmarks, including 91.9 PDMS on NAVSIM. The evaluation covers both planning performance and cross-dataset generalization.
  3. The paper systematically investigates attention mechanisms, video backbone choices, action model scaling, video supervision, and reinforcement learning. These experiments provide useful evidence for the effectiveness and flexibility of the framework.
Critical weaknesses
  1. The central idea of using future-video prediction during training while eliminating test-time imagination has already been explored by Fast-WAM. The proposed architecture and RL optimization largely combine existing techniques, making the independent methodological contribution somewhat limited.
  2. Although the method substantially reduces latency compared with imagination-based WAMs, the reported 518 ms inference latency on an A100 remains considerable for real-time autonomous driving. More comprehensive efficiency comparisons, including training costs and lightweight vision-backbone alternatives, would strengthen the practical claims.
  3. The zero-shot nuScenes results are competitive but do not consistently outperform existing WAM baselines. Moreover, the paper primarily relies on open-loop and pseudo-simulation benchmarks. Stronger out-of-distribution and long-horizon interactive evaluations would better establish the generalization benefits of the learned world representations.
Questions for authors
  1. How does SimWAM fundamentally differ from Fast-WAM beyond its adaptation to autonomous driving and the integration of existing RL techniques?
  2. Can the authors demonstrate that action-relevant world dynamics are learned, rather than simply obtaining stronger visual representations through future-video supervision? Comparisons against non-generative temporal auxiliary objectives would be valuable.
  3. Can the authors provide a more comprehensive performance-efficiency analysis and evaluate robustness under challenging distribution shifts or fully interactive closed-loop driving scenarios?
2 comments
AuthorsOpenReview ↗

Response to Reviewer qab8 1/2

We thank the reviewer for the careful evaluation on our work. We address the raised points in detail below.

To W1: Differences from existing methods

R: We appreciate the reviewer raising this point. SimWAM differs from existing methods in two aspects. First, the training strategy differs. Fast-WAM uses imitation learning only, targeting task completion in robot manipulation. In contrast, SimWAM focuses on autonomous driving, which requires the trajectory to remain safe and reasonable at all times. However, imitation learning only fits the logged demonstrations without any explicit safety signal. We therefore introduce RL with a driving reward that penalizes unsafe behaviors such as collisions, directly guiding the policy toward safe planning. Second, our RL is specifically designed for driving trajectories. We apply the SDE only to the last three denoising steps, which preserves trajectory plausibility and yields more diverse, informative rollouts, since the trajectory is most sensitive to noise there. We also anchor the policy to the frozen IL policy with per-axis weights that constrain the longitudinal axis more loosely than the lateral and heading axes, so RL improves progress while keeping lateral motion stable. We have discussed these differences in the revised manuscript.

To W2: Latency, training cost and lightweight backbones

R: Thank you for this practical perspective. Following your suggestion, we conduct two experiments on a single A100 GPU. First, we measure the latency of different lightweight video experts in Table 7. Second, we vary the denoising steps of Wan2.2-5B. The results are as follows:

Video expert (Table 7)Params (video)Compression, VAE + patch (T×H×W)PDMS IL (RL)InferenceVAE encoding (ms)Video DiT on current frame (ms)Per denoising step (ms)Denoising, 10 steps (ms)VAE decoding (ms)Others (ms)Total (ms)Peak memory (GiB)
LTX-Video1.9B8×32×3288.7 (90.9)Action-only19.651.448.7486.8–1.5559.17.5
Joint19.5–97.7977.736.449.51083.17.7
Wan2.1-1.3B1.3B4×16×1690.2 (91.8)Action-only21.944.244.7448.1–9.4519.94.8
Joint22.2–123.41234.4255.745.71557.96.7
Cosmos-Predict2.52.1B4×16×1690.4 (92.4)Action-only21.958.660.3603.3–1.5683.16.3
Joint22.2–152.81528.4256.226.21833.08.5
Wan2.2-5B (ours)5.0B4×32×3290.3 (91.9)Action-only14.344.444.7447.7–9.5518.012.9
Joint14.6–89.7896.8309.945.41266.715.5
Denoising stepsPDMSLatency (ms)
168.9115
290.2125
1090.3518
2090.2968

Since the video expert of SimWAM runs only once on the current frame, our action-only inference takes 518 ms, compared with 1266.7 ms for joint inference with future-frame generation. Notably, with only 2 denoising steps, SimWAM achieves 90.2 PDMS at 125 ms, already surpassing world-model planners with much higher latency in Fig. 1 (e.g., DriveWAM and DriveLaW). We use 10 steps by default for a fair comparison, as commonly adopted by other methods. In addition, a lower parameter count does not guarantee a lower latency, since the latency also depends on the compression ratio of the backbone, which determines the number of visual tokens. For instance, Wan2.1-1.3B has fewer parameters than Wan2.2-5B but takes longer in joint inference (1557.9 vs. 1266.7 ms) due to its lower compression ratio.

Regarding the training cost, SimWAM is trained on 48 A100 GPUs for about 18 hours. We have added the these analysis in the revised manuscript.

To W3: Out-of-distribution and closed-loop evaluation

R: Thank you for this constructive suggestion. Following your suggestion, we further evaluate SimWAM zero-shot on HUGSIM, a photo-realistic closed-loop simulator with 436 out-of-distribution scenarios. We adopt the IL weights trained on NAVSIM and report the overall Route Completion (RC) and HUGSIM Driving Score (HD-Score). The results are as follows:

Method (HUGSIM)RC ↑HD-Score ↑
VAD29.112.7
LTF37.722.4
DriveWAM41.925.4
SimWAM (ours)44.027.5

On HUGSIM, SimWAM achieves the highest overall RC and HD-Score among the compared methods. More importantly, compared with DriveWAM, an imagine-then-act WAM built on the same video expert, SimWAM improves both the overall RC and HD-Score by 2.1 points without generating any future frames at inference. This demonstrates that SimWAM achieves reliable closed-loop driving in out-of-distribution scenarios without explicit future-frame generation. We have added the HUGSIM results to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer qab8 2/2

To Q1: Differences from existing methods

R: Thank you for this question. As discussed in W1, our SimWAM differs from Fast-WAM in two aspects. First, the training strategy differs. Fast-WAM uses imitation learning only for robot manipulation, whereas our SimWAM targets autonomous driving, which requires the trajectory to remain safe and reasonable at all times, and therefore introduces RL with a driving reward that penalizes unsafe behaviors such as collisions. Second, our RL is specifically designed for driving trajectories. It applies the SDE only to the last three denoising steps and anchors the policy to the IL policy with a looser constraint on the longitudinal axis, allowing RL to improve progress while keeping the safety-critical lateral motion stable.

To Q2: Whether action-relevant dynamics are learned

R: We are grateful for this valuable question. Following your suggestion, we add two experiments. The first pretrains the video DiT with future-video generation and then freezes it when training the planner, and the second replaces future-frame generation with V-JEPA-style regression of future latents. The results are as follows:

IDVideo Expert TrainingFuture-Video SupervisionPDMS
1––86.5
2Joint–88.4
N1 (new)Two-stageGenerative89.1
N2 (new)JointFeature alignment (V-JEPA-style)89.7
3 (ours)JointGenerative90.3

Both the two-stage model (89.1) and V-JEPA-style alignment (89.7) outperform action-only training (88.4), and our joint generative co-training reaches the best 90.3, surpassing the two-stage model under the same objective (89.1). Together with the probes in Fig. 3, which show improved prediction of surrounding-vehicle motion and ego trajectories from the current-frame features, these results indicate that our co-training encourages representations containing planning-relevant dynamics information, rather than merely stronger visual features.

To Q3: Efficiency and robustness analysis

R: We thank you for this constructive question. Regarding the performance-efficiency analysis, as discussed in W2, SimWAM achieves 90.2 PDMS at 125 ms with only 2 denoising steps, already surpassing world-model planners with much higher latency in Fig. 1. Moreover, a lower parameter count does not guarantee a lower latency, as the latency also depends on the compression ratio of the backbone.

Regarding the closed-loop evaluation, as discussed in W3, we evaluate the NAVSIM-trained SimWAM zero-shot on 436 HUGSIM scenarios. It achieves the highest RC and HD-Score among the compared methods, surpassing the imagine-then-act DriveWAM by 2.1 points on both without generating future frames.

Thank you again for this valuable question. We have added these analyses and closed-loop results to the revised manuscript.

We hope these responses address the your concerns, and we would be glad to discuss any remaining questions.

Reviewer_UybaOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimWAM shows in a driving WAM based on a pre-trained video model, future video generation only needs to serve as auxiliary supervision during training by using an isolated attention mask to allow action tokens to only look at the current frame and not generate future frames at all during inference, planning performance comparable to the imagine-then-act scheme can be achieved, with latency reduced to about one-third; after adding GRPO-style RL, a PDMS of 91.9 is achieved on NAVSIM.

Critical strengths

The paper was evaluated on NAVSIM, WOD-E2E, and PhysicalAI-AV, and zero-shot transfer to nuScenes was also tested. Comparisons across different video backbones show that the quality of the video prior affects the final performance (Table 7). The action expert expanded from 0.21B to 1.02B, the PDMS improved by only 0.4, so the primary bottleneck likely remains on the video generation side, and that expanding the planning head yields limited benefits (Table 8). The improvement gained from extending the prediction time is also greater than that from increasing frame density (Table 12). Table 6 compares isolated masks, bidirectional masks, and action→video masks within the same model. While their planning performance is similar, there is a significant difference in latency, indicating that generating future frames during testing does not yield additional planning benefits. On PhysicalAI-AV, SimWAM achieved better results than the 13B DriveWAM with only 6B parameters, with ADE@4s values of 0.69 and 0.83, respectively.

Critical weaknesses
  1. Fast-WAM is a very similar approach for robotic tasks: using Wan2.2-TI2V-5B as the video backbone, two experts are jointly trained via shared attention and structured masks; video predictions are retained during training, while future tokens are removed during inference to directly output actions. SimWAM’s overall framework is very similar to Fast-WAM’s, but the relationship to Fast-WAM is not explained. The RL component also adds a BC anchoring term to the SDE sampling of Flow-GRPO, and the practice of using SDE only in certain denoising steps has precedents such as MixGRPO. The paper needs to clarify what exactly it adds compared to these works and why these designs are necessary. Otherwise, it is difficult to determine whether its main contribution is the transfer of existing WAM methods to autonomous driving or the proposal of a new WAM method.

  2. Table 1 and Fig. 1 use the 91.9 score achieved after RL, but compare it to other baselines trained only with IL. Furthermore, world model planners trained with RL, such as WorldRFT and DreamerAD, were already available prior to the paper’s submission deadline, yet the paper did not include them in the comparison.

Questions for authors
  1. Please explain how SimWAM’s video backbone, dual-expert attention interface, isolation mask structure, and joint training objective differ from those of Fast-WAM.

  2. Consider to add an oracle test on the bidirectional mask model: during inference, replace the generated frames with actual future frames.

  3. Table 6 is missing “future frames consider actions, but actions do not consider future frames,” which allows the video branch to be action-conditional without increasing inference latency. Try to include this ablation and report the PDMS and FVD results.

  4. Distinguish between the IL-only method and the method using RL in Table 1 and Fig. 1.

  5. For RL part: (a) What is the specific form of the reward? (b) What exactly does RL learn? Since the gains are almost entirely in EP, is this equivalent to simply extending the trajectory? (c) Is RL a necessary component of the method? if yes, why are no results reported for the RL-enhanced model on NAVSIM-v2 navtest and navhard?

  6. During training, the current frame is in the clean condition; during inference, the video DiT processes only the current frame. Please explain what the time step is for the video DiT during inference, whether token-wise time steps are used during training to fix the current frame at t=0, and whether there is any distribution drift between training and inference.

  7. Are there any omissions in the discussion of related work? For example, Metis also decouples the video generation expert from the action expert and supports action-only inference (0.17 s).

3 comments
AuthorsOpenReview ↗

Response to Reviewer Uyba 1/3

We thank the reviewer for the valuable suggestions. We have provided the requested comparisons with existing methods and included additional experiments.

To W1: Relationship to Fast-WAM and novelty of the RL design

R: We appreciate the reviewer raising this point. Regarding Fast-WAM, SimWAM differs in two aspects. First, the training strategy differs. Fast-WAM uses imitation learning only, targeting task completion in robot manipulation. In contrast, SimWAM focuses on autonomous driving, which requires the trajectory to remain safe and reasonable at all times. However, imitation learning only fits the logged demonstrations without any explicit safety signal. We therefore introduce RL with a driving reward, directly guiding the policy toward safe and reasonable planning. Second, the motivation differs. Fast-WAM aims to examine whether explicit future-frame prediction is necessary for robot manipulation, whereas SimWAM aims to establish a simple yet solid baseline for autonomous driving by exploiting the dynamics prior of pretrained video models.

Regarding the RL design, our RL stage differs from Flow-GRPO and MixGRPO in two aspects. First, the domain is different. Flow-GRPO and MixGRPO are developed for visual generation and focus on visual quality, whereas SimWAM optimizes physical trajectories and constrains their physical plausibility. Second, the purpose of the SDE window is different. MixGRPO restricts the SDE to a window of denoising steps to accelerate convergence and shape the image layout, whereas SimWAM confines the SDE to the last denoising steps, to which the trajectory is more sensitive, yielding more diverse trajectories and more informative rewards.

Considering that this is an early work introducing a simple yet effective world-action modeling approach validated across multiple autonomous driving benchmarks, we believe it still brings useful insights to the community. We have discussed the relationship with Fast-WAM and the RL design in the revised manuscript.

To W2: IL/RL comparison and missing RL-based world models

R: We thank the reviewer for the helpful comments. We compare SimWAM with the RL-based world-model planners WorldRFT and DreamerAD as below:

MethodReferenceSensorsTrainingPDMS
WorldRFTAAAI'263×CRL87.8
DreamerADECCV'261×CRL88.7
SimWAM-IL (ours)–1×CIL90.3
SimWAM-RL (ours)–1×CRL91.9

Under RL, SimWAM outperforms the world-model planners WorldRFT and DreamerAD by 4.1 and 3.2 PDMS, respectively. Notably, even with imitation learning alone, SimWAM already surpasses both, further supporting the effectiveness of our method. We have added WorldRFT and DreamerAD to Table 1 and marked the training paradigm (IL or RL) of each method in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Uyba 2/3

To Q1: Differences from existing methods

R: Thank you for this question. As discussed in response W1, SimWAM differs in two aspects. First, the training strategy differs. Fast-WAM uses imitation learning only for robot manipulation, whereas SimWAM targets autonomous driving, which requires the trajectory to remain safe and reasonable at all times, and therefore further introduces RL with a driving reward to directly guide the policy toward safe and reasonable planning. Second, the motivation differs. Fast-WAM aims to examine whether explicit future-frame prediction is necessary for robot manipulation, whereas SimWAM aims to establish a simple yet solid baseline for autonomous driving by exploiting the dynamics prior of pretrained video models.

To Q2: Oracle test with ground-truth future frames

R: Thank you for this helpful suggestion. Following your advice, we conduct an oracle test on the bidirectional-mask model of Table 6. The results are as follows:

Future inputPDMS
Bidirectional, generated future frames (Table 6)90.2
Bidirectional, ground-truth future frames92.7

Replacing the generated futures with ground-truth ones raises PDMS from 90.2 to 92.7, indicating that accurate future frames hold promising potential for further improving planning. Notably, obtaining ground-truth future frames is infeasible in most driving scenarios, since they are only available after the ego vehicle has driven.

To Q3: Action-conditioned video mask

R: Thank you for this helpful suggestion. Following your suggestion, we train this configuration and compare it with our isolated mask. The results are as follows:

MaskAction reads futureFuture reads actionPDMSFVD ↓
Isolated✗✗90.336.3
Action-conditioned video✗✓90.135.8

The two masks achieve comparable planning performance (90.3 vs. 90.1 PDMS), while conditioning the future frames on the actions slightly improves FVD (35.8 vs. 36.3). Considering that the isolated mask already enables the model to learn the driving dynamics that facilitate planning, it is sufficient for SimWAM. We have added this configuration to Table 6 of the revised manuscript.

To Q4: Distinguishing IL and RL results

R: Thank you for this suggestion. We have distinguished the IL and RL results in Table 1 and Fig. 1 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Uyba 3/3

To Q5: Reward form, learned behavior and necessity of RL

R: Thank you for these points.

Regarding (a): We use a PDMS-based training reward containing NC, DAC, EP, TTC, and Comfort, with the EP weight increased from the official value of 5 to 10 during training. We have specified the reward in the revised manuscript.

Regarding (b): To test this hypothesis, we uniformly stretch the trajectories predicted by the IL model along the longitudinal direction by a factor α\alpha. The results are as follows:

MethodNCDACEPTTCCPDMS
IL98.698.084.095.8100.090.3
Stretched IL (α\alpha = 1.05)98.092.080.395.0100.084.7
Stretched IL (α\alpha = 1.10)97.591.081.089.0100.082.7
Stretched IL (α\alpha = 1.20)97.588.080.388.0100.080.3
RL98.599.186.596.199.891.9

Stretching the IL trajectories does not increase EP and instead degrades DAC and TTC, with PDMS dropping from 84.7 to 80.3 as α\alpha increases, all even below the IL result. In contrast, RL raises EP from 84.0 to 86.5 while also improving DAC and TTC, reaching 91.9 PDMS. This demonstrates that RL does not simply extend the trajectory, but learns a safety-aware policy that pursues greater ego progress while preserving drivable-area compliance and safety margins.

Regarding (c): We would like to clarify that RL is a necessary component of SimWAM. Autonomous driving requires the trajectory to remain safe and reasonable at all times. However, imitation learning only fits the logged demonstrations without any explicit safety signal. We therefore introduce RL with a driving reward to directly guide the policy toward safe and reasonable planning, which improves PDMS from 90.3 to 91.9. Sorry for missing the RL results on NAVSIM-v2. We report them on navtest and navhard as follows:

Stagenavtest EPDMSnavhard EPDMS
IL90.237.6
RL90.838.8

Similar to the PDMS improvement on NAVSIM-v1, RL also raises EPDMS on NAVSIM-v2 from 90.2 to 90.8 on navtest and from 37.6 to 38.8 on navhard. We have added the NAVSIM-v2 RL results to the revised manuscript.

To Q6: Video DiT timestep and train–inference drift

R: Thank you for the careful examination of our implementation. We clarify these details as follows.

(1) Token-wise timesteps. We indeed adopt token-wise timesteps during training. For the video DiT, the current frame is fed as its clean latent at τ = 0, while the future frames share a sampled noise timestep τ ∈ (0, 1]. At inference, the video DiT processes only the current-frame tokens at τ = 0.

(2) Distribution drift. Our design introduces no such drift. Under the isolated mask, the current-frame tokens never attend to the future-frame tokens during training, so they receive the same inputs and attention context at training and inference. To verify this, we compare our action-only inference with joint inference, which also generates the future frames as in training:

Inference modePDMS
Joint (with future-frame generation)90.3
Action-only90.3

The two inference modes yield identical results, confirming that there is no distribution drift between training and inference.

To Q7: Comparison with Metis

R: Thank you for this question. We would like to first clarify that Metis is a concurrent work also under review at ICLR 2027. We summarize the key differences between our SimWAM and Metis as follows:

  • Training strategy. Metis trains its planner with imitation learning (IL) only, which confines the policy to the logged demonstrations and only indirectly reflects driving quality. SimWAM instead introduces an RL stage that directly optimizes the driving reward, allowing the policy to explore beyond the demonstrations and align its planning with safety, compliance and progress.
  • Attention mask. Metis lets its video expert attend to the action tokens, whereas the video and action experts of SimWAM are mutually isolated.
  • Performance and efficiency. SimWAM outperforms Metis on both NAVSIM-v1 navtest and NAVSIM-v2 navhard even under the same IL setting. Notably, the 0.17 s latency of Metis is measured on an RTX 4090 GPU with 2 denoising steps, so for a fair comparison we also evaluate SimWAM under the same configuration, where it runs faster than Metis (145 ms vs. 170 ms), as shown in the table below.
MethodStepsLatency (ms)NAVSIM-v1 navtest PDMSNAVSIM-v2 navhard EPDMS
Metis217088.931.2
SimWAM (ours)214590.234.6

We hope these responses address your concerns, and we would be glad to discuss any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the AC

Dear Area Chair,

We sincerely thank you and all reviewers for the time and effort spent evaluating our submission and for the constructive feedback. For your convenience, we would like to briefly summarize the outcome of the rebuttal and discussion. Following the rebuttal, we are grateful that all three reviewers acknowledged the value of our work. We also appreciate the reviewers' recognition of SimWAM's simple yet effective design, efficient inference without future-video generation, and strong empirical results across multiple driving benchmarks, supported by extensive ablations and additional experiments.

Following the reviewers' valuable suggestions, we have carefully revised the manuscript, with all changes highlighted in blue. We sincerely appreciate your time and effort in evaluating our work. Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.