- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces DreamerAD, a framework designed to fine-tune end-to-end driving planners via reinforcement learning (RL) within a video world model. To address the computational bottleneck of multi-step denoising in video diffusion models during imagined rollouts, the paper proposes three main components:
- Shortcut Forcing World Model (SF-WM): Employs self-distillation with an EMA teacher to achieve one-step generation, drastically reducing latency (e.g., from 403.3ms to 30.2ms on Epona) while maintaining EPDMS performance.
- Autoregressive Dense Reward Model (AD-RM): Predicts 8 NAVSIM sub-metrics across 8 future steps by reading observed/imagined latents and trajectory prefixes, trained on filtered simulation candidates.
- Gaussian Vocabulary Sampling for GRPO: Samples candidates from a filtered trajectory vocabulary (from GTRS) weighted by a Gaussian distribution centered on policy outputs, optimized via a clipped GRPO objective with a BC loss.
Evaluated on Epona and Wan2.2 backbones, the method shows improvements on NAVSIM v1/v2 and zero-shot transfer on HUGSIM.
Critical strengths
-
Effective One-Step World Model Design: SF-WM successfully addresses the core efficiency bottleneck of video world model-based RL. Achieving one-step rollouts that match 16-step performance while drastically reducing latency on both a specialized driving architecture (Epona) and a general 5B video DiT (Wan2.2) is impressive and practically valuable.
-
Comprehensive Benchmarking and Cost Reporting: The paper evaluates across multiple benchmarks (NAVSIM v1, v2, and zero-shot HUGSIM) and provides useful analyses on training overhead, data scaling, ranking accuracy, and component-wise ablations.
Critical weaknesses
W1. The specific contribution of latent imagination is not isolated: AD-RM labels are derived from NAVSIM simulations of 256 candidates per training scene, and RL samples from this same filtered vocabulary, meaning exact simulator scores are available for every training candidate.
W2. Confounding gains from expert anchoring and behavioral cloning: The candidate vocabulary is heavily pre-filtered using ground-truth expert trajectories (, , ). The baseline comparisons (WorldRFT, Flow-GRPO) test unfiltered candidates against an AD-RM model trained solely on the filtered distribution, which likely subjects the reward model to out-of-distribution queries.
W3. Underspecified training dynamics and efficiency trade-offs: While inference latency is reduced, total training is heavily dominated by reward-model training (~1 week on 32 H20 GPUs vs. ~8 hours for RL), which partially offsets the claimed training efficiency.
Questions for authors
Q1. Simulator Score Baseline: Does RL run on the exact NAVSIM training scenes used to label AD-RM? If so, what EPDMS do you achieve if RL uses the direct NAVSIM simulator scorer as the reward on the same candidate set?
Q2. Ablating Latent Imagination: How does AD-RM perform without imagined latents (i.e., using history and trajectory prefixes only) in terms of ranking accuracy on navtest and downstream EPDMS/HUGSIM performance?
Q3. BC Control: What performance does a continued-BC-only baseline achieve under the same training schedule?
Q4. Implementation Details: How are step-wise rewards computed from NAVSIM's aggregate metrics? For Wan2.2, which specific parameters are updated by RL, and is the imagination world model kept frozen during policy optimization?
6 comments
Reward Comparisons, Sampling Controls, and Training Efficiency
W2/Q3: Expert filtering, out-of-distribution RM queries, and BC-only control
Matched BC-only control. This control examines whether continued supervised training can account for the improvement attributed to reward-guided policy optimization. We therefore compare BC-only continuation with full RL using the same initialization and update budget. Starting from the post-SF policy of Table 4, ID 6, this control continues training for 2 epochs using only the BC loss, matching the full RL budget.
Table E4. BC-only and full RL with matched initialization and training budgets.
| Training objective | Initial EPDMS | Final EPDMS ↑ |
|---|---|---|
| BC-only (matched budget) | 88.8 | 89.0 |
| Full RL | 88.8 | 91.4 |
Full RL achieves a final EPDMS 2.4 points higher than BC-only. Thus, additional BC updates do not account for the RL improvement under this matched budget.
Candidate filtering and RM generalization. We appreciate the concern that expert-based filtering may favor the vocabulary sampler, while candidates from other samplers may fall outside the RM’s training distribution. To examine this alongside expert-based filtering, we compare adaptations of the WorldRFT [3] and Flow-GRPO [4] sampling strategies with two vocabulary-sampling settings. The original supervised candidates were sampled from an 8192-entry parent vocabulary. The expanded RL sampling setting uses this full parent vocabulary, which contains both the original candidates and additional candidates not used for RM supervision.
Table E5. RM performance, candidate quality, and policy performance under different samplers.
| Candidate sampler | MAE ↓ | PA ↑ | Groups with max score >90 ↑ | Groups with all scores <50 ↓ | Mean candidate score ↑ | EPDMS ↑ |
|---|---|---|---|---|---|---|
| WorldRFT | 0.40 | 67.73% | 51.21% | 13.15% | 61.91 | 89.5 |
| Flow-GRPO | 0.38 | 68.92% | 57.32% | 11.80% | 65.40 | 90.1 |
| Vocabulary sampling (supervised subset) | 0.27 | 77.62% | 78.93% | 4.00% | 79.34 | 91.4 |
| Vocabulary sampling (8192-entry parent vocabulary) | 0.30 | 71.21% | 88.30% | 2.70% | 81.32 | 91.8 |
Candidate scores and EPDMS in Table E5 use a 0–100 scale. MAE measures the aggregated log-reward error. Ranking accuracy on WorldRFT and Flow-GRPO candidates is lower than on the original vocabulary candidates, but their mean candidate quality is also lower, with more groups containing only low-scoring trajectories. The dynamically discontinuous trajectories in Appendix D.5, Figure 9 provide corresponding qualitative evidence. With the full parent vocabulary, PA is lower than for the original supervised subset, yet high-scoring candidate coverage, mean candidate score, and final EPDMS all improve. This indicates that the RM remains useful for optimizing over additional unsupervised candidates.
Supervision within the filtered candidate pool. Across all 103,288 training scenes, with at most 256 original candidates per scene, 38.08% of candidates have zero scores and 32.51% have high scores (4 s trajectory score ≥90); 95.50% of valid scenes contain both zero- and high-scoring candidates. The expert-centered pool therefore retains substantial variation in trajectory quality and safety. These results support the practical value of extending the RM to unsupervised candidates within the same scenes. Taken together, the sampling results suggest that lower candidate quality also contributes to the weaker downstream performance of the WorldRFT and Flow-GRPO sampling variants.
Reward Comparisons, Sampling Controls, and Training Efficiency
W3: RM training cost and overall training efficiency
We thank the reviewer for highlighting the need to account for RM training when assessing overall efficiency, and apologize for not presenting the costs of both backbones together.
Training costs across world-model backbones. Table E6 compares the driving-specific Epona backbone with the general-purpose Wan2.2 backbone, showing how backbone choice and training-data volume affect RM training cost and downstream performance.
Table E6. Estimated RM training costs and downstream policy performance. All training times are estimates for 32 H20 GPUs under the respective training configurations.
| Backbone | RM training-scene fraction | Estimated RM training time | Downstream EPDMS ↑ |
|---|---|---|---|
| Epona | 20% | ~20 h | 87.5 |
| Epona | 100% | ~7 days | 87.8 |
| Wan2.2 | 20% | ~5 h | 91.2 |
| Wan2.2 | 100% | ~1 day | 91.4 |
Efficient future generation with Wan2.2. Epona generates future frames autoregressively, whereas Wan2.2 uses fourfold temporal compression and jointly generates future latent blocks, reducing the need for sequential frame-by-frame rollout. With our training configurations, the Wan2.2-based RM requires approximately one day on the full dataset.
Lower training cost with strong downstream performance. Using only 20% of the training scenes further reduces the estimated Wan2.2 RM training time to approximately five hours, while retaining nearly the same downstream performance: 91.2 versus 91.4 EPDMS. These results suggest that the general-purpose Wan2.2 backbone offers an efficient route to effective reward modeling, providing a practical foundation for our goal of building a reusable RM for subsequent policy optimization.
RL-stage cost and reward-generation speed. Our RL pipeline already uses SF to accelerate dense reward generation. AD-RM generates dense rewards online approximately 2.14× faster than simulator scoring (Table E1) and 16× faster than the model without SF.
Reward Comparisons, Sampling Controls, and Training Efficiency
We appreciate the reviewer’s questions about direct simulator rewards, the contribution of imagination, sampling and BC controls, and computational cost. These comments helped us strengthen the experimental comparisons and clarify the training procedure and computational trade-offs.
Thank you for suggesting a direct comparison with simulator rewards under a matched training budget.
Direct simulator-reward comparison. Yes. RL uses the same training scenes used for AD-RM labeling, and simulator labels are available for all RL candidates in the original vocabulary setting. To directly assess reward approximation, we compare ground-truth trajectory-level rewards, ground-truth dense rewards, and AD-RM rewards. The three runs share the same initial policy, candidate sampling, training data, and two-epoch update budget; only the reward source and temporal density differ. Trajectory-level rewards score the complete 4 s trajectory, whereas dense rewards score each of its eight prefixes. Dense simulator rewards improve on final-trajectory rewards, and AD-RM further increases the final EPDMS (Table E1).
Table E1. RL with simulator rewards and AD-RM rewards.
Online reward-generation efficiency. AD-RM provides approximately 2.14× faster online dense reward generation on a single H20 with batch size 1. The timings in Table E1 include both future-latent generation and RM prediction for AD-RM, and scoring all eight trajectory prefixes for the simulator.
Why learned rewards may help. AD-RM's continuous predictions may provide smoother relative reward signals for policy optimization. Some simulator safety metrics are binary or discrete, whereas the RM outputs continuous scores; these predictions can change within-group reward differences while retaining useful candidate rankings. On a fixed set of 1,024 training scenes, with only the group mean subtracted, the mean absolute reward deviation decreases from 0.1919 to 0.1150, a reduction of 40.1%. This diagnostic supports the observation of smaller within-group reward differences and provides a possible explanation for smoother optimization signals. We also observed larger training-time EPDMS fluctuations when using ground-truth rewards.
Previous preference-based RL studies have likewise observed learned rewards outperforming direct environment rewards on some tasks under specific feedback settings, with reward shaping or the encoding of longer-term information offered as possible explanations [1] [2]. Our experiment directly demonstrates the performance and online reward-generation benefits of AD-RM under a matched training budget.