acceptodds
Under review as a conference paper at ICLR 2027

DreamerAD: Efficient Reinforcement Learning via Latent World Model for Autonomous Driving

Abstract

Training reinforcement learning (RL) policies through real-world interaction is costly and risky. Video-generation-based world models can provide realistic video rollouts in place of real-world interaction, but multi-step denoising and RGB decoding make these imagined rollouts computationally expensive. We introduce DreamerAD, a framework for efficient reinforcement learning through latent-space imagination for autonomous driving. Our approach leverages denoised latent features from video generation models through three key mechanisms: (1) shortcut forcing that distills multi-step flow updates into low-step latent rollouts, accelerating world model inference by up to , (2) an autoregressive dense reward model that predicts step-wise rewards directly from latent representations, providing temporally dense feedback for policy optimization, and (3) Gaussian vocabulary sampling for GRPO that constrains exploration to physically plausible trajectories. DreamerAD achieves 91.4 EPDMS on NAVSIM v2 and 55.0 HD-Score on zero-shot HUGSIM, establishing state-of-the-art performance across both benchmarks. Across both the driving-specific Epona world model and the general-purpose Wan2.2 video generation backbone, reinforcement learning through latent-space imagination improves zero-shot closed-loop driving performance on HUGSIM.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Reward Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2eJYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DreamerAD, a framework designed to fine-tune end-to-end driving planners via reinforcement learning (RL) within a video world model. To address the computational bottleneck of multi-step denoising in video diffusion models during imagined rollouts, the paper proposes three main components:

  1. Shortcut Forcing World Model (SF-WM): Employs self-distillation with an EMA teacher to achieve one-step generation, drastically reducing latency (e.g., from 403.3ms to 30.2ms on Epona) while maintaining EPDMS performance.
  2. Autoregressive Dense Reward Model (AD-RM): Predicts 8 NAVSIM sub-metrics across 8 future steps by reading observed/imagined latents and trajectory prefixes, trained on filtered simulation candidates.
  3. Gaussian Vocabulary Sampling for GRPO: Samples candidates from a filtered trajectory vocabulary (from GTRS) weighted by a Gaussian distribution centered on policy outputs, optimized via a clipped GRPO objective with a BC loss.

Evaluated on Epona and Wan2.2 backbones, the method shows improvements on NAVSIM v1/v2 and zero-shot transfer on HUGSIM.

Critical strengths
  1. Effective One-Step World Model Design: SF-WM successfully addresses the core efficiency bottleneck of video world model-based RL. Achieving one-step rollouts that match 16-step performance while drastically reducing latency on both a specialized driving architecture (Epona) and a general 5B video DiT (Wan2.2) is impressive and practically valuable.

  2. Comprehensive Benchmarking and Cost Reporting: The paper evaluates across multiple benchmarks (NAVSIM v1, v2, and zero-shot HUGSIM) and provides useful analyses on training overhead, data scaling, ranking accuracy, and component-wise ablations.

Critical weaknesses

W1. The specific contribution of latent imagination is not isolated: AD-RM labels are derived from NAVSIM simulations of 256 candidates per training scene, and RL samples from this same filtered vocabulary, meaning exact simulator scores are available for every training candidate.

W2. Confounding gains from expert anchoring and behavioral cloning: The candidate vocabulary is heavily pre-filtered using ground-truth expert trajectories (∣Δx∣≤10 m\vert{}\Delta x\vert{} \le 10\text{ m}, ∣Δy∣≤5 m\vert{}\Delta y\vert{} \le 5\text{ m}, Δθ≤20∘\Delta \theta \le 20^\circ). The baseline comparisons (WorldRFT, Flow-GRPO) test unfiltered candidates against an AD-RM model trained solely on the filtered distribution, which likely subjects the reward model to out-of-distribution queries.

W3. Underspecified training dynamics and efficiency trade-offs: While inference latency is reduced, total training is heavily dominated by reward-model training (~1 week on 32 H20 GPUs vs. ~8 hours for RL), which partially offsets the claimed training efficiency.

Questions for authors

Q1. Simulator Score Baseline: Does RL run on the exact NAVSIM training scenes used to label AD-RM? If so, what EPDMS do you achieve if RL uses the direct NAVSIM simulator scorer as the reward on the same candidate set?

Q2. Ablating Latent Imagination: How does AD-RM perform without imagined latents (i.e., using history and trajectory prefixes only) in terms of ranking accuracy on navtest and downstream EPDMS/HUGSIM performance?

Q3. BC Control: What performance does a continued-BC-only baseline achieve under the same training schedule?

Q4. Implementation Details: How are step-wise rewards computed from NAVSIM's aggregate metrics? For Wan2.2, which specific parameters are updated by RL, and is the imagination world model kept frozen during policy optimization?

6 comments
AuthorsOpenReview ↗

Reward Comparisons, Sampling Controls, and Training Efficiency

We appreciate the reviewer’s questions about direct simulator rewards, the contribution of imagination, sampling and BC controls, and computational cost. These comments helped us strengthen the experimental comparisons and clarify the training procedure and computational trade-offs.

W1/Q1: Direct simulator rewards on the same scenes and candidates

Thank you for suggesting a direct comparison with simulator rewards under a matched training budget.

Direct simulator-reward comparison. Yes. RL uses the same training scenes used for AD-RM labeling, and simulator labels are available for all RL candidates in the original vocabulary setting. To directly assess reward approximation, we compare ground-truth trajectory-level rewards, ground-truth dense rewards, and AD-RM rewards. The three runs share the same initial policy, candidate sampling, training data, and two-epoch update budget; only the reward source and temporal density differ. Trajectory-level rewards score the complete 4 s trajectory, whereas dense rewards score each of its eight prefixes. Dense simulator rewards improve on final-trajectory rewards, and AD-RM further increases the final EPDMS (Table E1).

Table E1. RL with simulator rewards and AD-RM rewards.

Reward sourceEPDMS ↑Reward generation latency (ms/trajectory) ↓
Simulator: trajectory-level reward (4 s)90.4-
Simulator: dense reward91.0360.74
AD-RM: dense reward91.4168.49

Online reward-generation efficiency. AD-RM provides approximately 2.14× faster online dense reward generation on a single H20 with batch size 1. The timings in Table E1 include both future-latent generation and RM prediction for AD-RM, and scoring all eight trajectory prefixes for the simulator.

Why learned rewards may help. AD-RM's continuous predictions may provide smoother relative reward signals for policy optimization. Some simulator safety metrics are binary or discrete, whereas the RM outputs continuous scores; these predictions can change within-group reward differences while retaining useful candidate rankings. On a fixed set of 1,024 training scenes, with only the group mean subtracted, the mean absolute reward deviation decreases from 0.1919 to 0.1150, a reduction of 40.1%. This diagnostic supports the observation of smaller within-group reward differences and provides a possible explanation for smoother optimization signals. We also observed larger training-time EPDMS fluctuations when using ground-truth rewards.

Previous preference-based RL studies have likewise observed learned rewards outperforming direct environment rewards on some tasks under specific feedback settings, with reward shaping or the encoding of longer-term information offered as possible explanations [1] [2]. Our experiment directly demonstrates the performance and online reward-generation benefits of AD-RM under a matched training budget.

AuthorsOpenReview ↗

Reward Comparisons, Sampling Controls, and Training Efficiency

W2/Q3: Expert filtering, out-of-distribution RM queries, and BC-only control

Matched BC-only control. This control examines whether continued supervised training can account for the improvement attributed to reward-guided policy optimization. We therefore compare BC-only continuation with full RL using the same initialization and update budget. Starting from the post-SF policy of Table 4, ID 6, this control continues training for 2 epochs using only the BC loss, matching the full RL budget.

Table E4. BC-only and full RL with matched initialization and training budgets.

Training objectiveInitial EPDMSFinal EPDMS ↑
BC-only (matched budget)88.889.0
Full RL88.891.4

Full RL achieves a final EPDMS 2.4 points higher than BC-only. Thus, additional BC updates do not account for the RL improvement under this matched budget.

Candidate filtering and RM generalization. We appreciate the concern that expert-based filtering may favor the vocabulary sampler, while candidates from other samplers may fall outside the RM’s training distribution. To examine this alongside expert-based filtering, we compare adaptations of the WorldRFT [3] and Flow-GRPO [4] sampling strategies with two vocabulary-sampling settings. The original supervised candidates were sampled from an 8192-entry parent vocabulary. The expanded RL sampling setting uses this full parent vocabulary, which contains both the original candidates and additional candidates not used for RM supervision.

Table E5. RM performance, candidate quality, and policy performance under different samplers.

Candidate samplerMAE ↓PA ↑Groups with max score >90 ↑Groups with all scores <50 ↓Mean candidate score ↑EPDMS ↑
WorldRFT0.4067.73%51.21%13.15%61.9189.5
Flow-GRPO0.3868.92%57.32%11.80%65.4090.1
Vocabulary sampling (supervised subset)0.2777.62%78.93%4.00%79.3491.4
Vocabulary sampling (8192-entry parent vocabulary)0.3071.21%88.30%2.70%81.3291.8

Candidate scores and EPDMS in Table E5 use a 0–100 scale. MAE measures the aggregated log-reward error. Ranking accuracy on WorldRFT and Flow-GRPO candidates is lower than on the original vocabulary candidates, but their mean candidate quality is also lower, with more groups containing only low-scoring trajectories. The dynamically discontinuous trajectories in Appendix D.5, Figure 9 provide corresponding qualitative evidence. With the full parent vocabulary, PA is lower than for the original supervised subset, yet high-scoring candidate coverage, mean candidate score, and final EPDMS all improve. This indicates that the RM remains useful for optimizing over additional unsupervised candidates.

Supervision within the filtered candidate pool. Across all 103,288 training scenes, with at most 256 original candidates per scene, 38.08% of candidates have zero scores and 32.51% have high scores (4 s trajectory score ≥90); 95.50% of valid scenes contain both zero- and high-scoring candidates. The expert-centered pool therefore retains substantial variation in trajectory quality and safety. These results support the practical value of extending the RM to unsupervised candidates within the same scenes. Taken together, the sampling results suggest that lower candidate quality also contributes to the weaker downstream performance of the WorldRFT and Flow-GRPO sampling variants.

AuthorsOpenReview ↗

Reward Comparisons, Sampling Controls, and Training Efficiency

W3: RM training cost and overall training efficiency

We thank the reviewer for highlighting the need to account for RM training when assessing overall efficiency, and apologize for not presenting the costs of both backbones together.

Training costs across world-model backbones. Table E6 compares the driving-specific Epona backbone with the general-purpose Wan2.2 backbone, showing how backbone choice and training-data volume affect RM training cost and downstream performance.

Table E6. Estimated RM training costs and downstream policy performance. All training times are estimates for 32 H20 GPUs under the respective training configurations.

BackboneRM training-scene fractionEstimated RM training timeDownstream EPDMS ↑
Epona20%~20 h87.5
Epona100%~7 days87.8
Wan2.220%~5 h91.2
Wan2.2100%~1 day91.4

Efficient future generation with Wan2.2. Epona generates future frames autoregressively, whereas Wan2.2 uses fourfold temporal compression and jointly generates future latent blocks, reducing the need for sequential frame-by-frame rollout. With our training configurations, the Wan2.2-based RM requires approximately one day on the full dataset.

Lower training cost with strong downstream performance. Using only 20% of the training scenes further reduces the estimated Wan2.2 RM training time to approximately five hours, while retaining nearly the same downstream performance: 91.2 versus 91.4 EPDMS. These results suggest that the general-purpose Wan2.2 backbone offers an efficient route to effective reward modeling, providing a practical foundation for our goal of building a reusable RM for subsequent policy optimization.

RL-stage cost and reward-generation speed. Our RL pipeline already uses SF to accelerate dense reward generation. AD-RM generates dense rewards online approximately 2.14× faster than simulator scoring (Table E1) and 16× faster than the model without SF.

Reviewer_TGHhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses an important computational bottleneck in using video-based world models for reinforcement learning in autonomous driving. The integration of shortcut-based generation, latent reward prediction, and constrained trajectory exploration is practically meaningful. Experimental results show improvements over the Epona backbone, and the released implementation improves reproducibility.

Critical strengths

See the "Paper Summary"

Critical weaknesses

The methodological novelty largely stems from combining existing components. More importantly, the experiments do not sufficiently isolate the necessity of imagination-based world modeling from reward approximation and constrained policy refinement. Quantitative evaluations of counterfactual world-model fidelity and reward prediction accuracy are missing. The reward formulation and policy likelihood computation also require clarification.

Questions for authors

See the "Critical Weaknesses"

2 comments
AuthorsOpenReview ↗

Methodological Contribution and Evidence for Future Imagination

W/Q: Methodological novelty, the role of imagination, and formulation details

Novelty beyond combining existing components

DreamerAD directly uses denoised latents from a video-generation world model for trajectory-conditioned future imagination and autoregressive dense reward prediction, then optimizes an end-to-end driving policy using the predicted rewards.

ReCogDrive [1] and DiffusionDriveV2 [2] use trajectory-level NAVSIM rewards to guide policy optimization. SimVTA [3] also uses trajectory-level NAVSIM rewards for GRPO, with additional supervision from trajectory-matched videos rendered with 3DGS.

Table T1. Performance on NAVSIM and HUGSIM.

MethodNAVSIM V1 PDMS ↑NAVSIM V2 EPDMS ↑HUGSIM HD-Score ↑
ReCogDrive [1]90.883.6-
DiffusionDriveV2 [2]91.287.5-
SimVTA [3]91.388.238.8
DreamerAD92.391.454.96

Isolating imagination from reward approximation and policy refinement

Reward prediction accuracy and safety. We compare a context-and-trajectory RM with an RM that additionally uses imagined future latents on all 12,146 navtest scenes. Both RMs are trained for 2 epochs. Each scene has 8 trajectories randomly sampled from the vocabulary and 8 RL candidates generated using the RL sampling procedure from the reported checkpoint with 91.4 EPDMS.

Table T2. Reward prediction on navtest with different RM inputs.

RM inputRandom-vocabulary MAE ↓RL-candidate MAE ↓Random-vocabulary PA ↑RL-candidate PA ↑False-safe selection rate ↓
Context + trajectory0.720.3178.83%68.23%6.90%
Context + trajectory + imagined future0.480.2782.64%77.62%3.71%

MAE measures error in the aggregated log reward, and PA measures pairwise ranking accuracy. The false-safe selection rate is the fraction of all evaluated scenes where the RM's top-ranked trajectory has predicted NC, DAC, and TTC ≥0.5 at 4 s, but at least one corresponding simulator value is 0.

Table T2 shows lower reward prediction error and higher ranking accuracy with future imagination on both candidate distributions. On RL candidates, PA improves by 9.39 percentage points, accompanied by a lower false-safe selection rate.

Downstream policy improvement. We also compare the final policy scores after RL with each RM (Table T3).

Table T3. Downstream policy performance with different RM inputs.

RM inputEPDMS ↑HD-Score ↑
Context + trajectory91.051.59
Context + trajectory + imagined future91.454.96

Table T3 shows that adding future imagination improves the final policy's EPDMS and HD-Score by 0.4 and 3.37 points, respectively, over the context-only variant. Together with the improved candidate ranking and lower false-safe selection rate, these downstream gains support the value of imagined future states for reward-guided policy optimization.

AuthorsOpenReview ↗

Methodological Contribution and Evidence for Future Imagination

Counterfactual world-model fidelity

Thank you for raising the question of counterfactual world-model fidelity. Ground-truth future observations matched to counterfactual candidate trajectories are unavailable, so we assess trajectory-conditioned future generation qualitatively. Figure 1 and the additional visualizations now included in the supplementary material show imagined futures under different candidate trajectories, allowing inspection of intermediate states and potential accident sequences.

Reward formulation and dense reward computation

We independently score the eight trajectory prefixes ending at 0.5, 1.0, …, 4.0 s with the simulator, accumulating collision-related labels up to each prefix endpoint. AD-RM learns to predict eight metrics at each horizon using weighted BCE, then combines the metrics and applies temporal weighting according to Eq. (9):

\prod_{k\in S}(p_k^j)^{w_k} \cdot\sum_{k\in P}w_kp_k^j \right], \qquad R=\sum_{j=1}^{T}\lambda_j r_j.$$ Here, $p_k^j$ is the prediction for metric $k$ at horizon $j$, and $w_k$ and $\lambda_j$ are the metric and temporal weights, respectively; $S=\{NC,DAC,DDC,TLC\}$, $P=\{EP,TTC,LK,HC\}$, and $T=8$. ### **Policy likelihood used for RL updates** RL updates use a Gaussian proxy density centered on the policy's predicted trajectory: $$\widetilde{\pi}_{\theta}(\tau) = \mathcal{N}(\tau;\tau_{\mathrm{act},\theta},\sigma^2 I).$$ For the same candidate trajectory, the proxy density ratio between the current and old policies is: $$\rho_{\theta}(\tau) = \frac{\widetilde{\pi}_{\theta}(\tau)}{\widetilde{\pi}_{\theta_{\mathrm{old}}}(\tau)}.$$ ### **Summary Response** We have clarified how DreamerAD uses trajectory-conditioned video latents for dense reward prediction and policy optimization. The additional RM comparisons provide quantitative evidence on reward prediction and downstream policy performance, while the supplementary visualizations provide qualitative examples of imagined futures under alternative trajectories. We also explain dense reward construction and the Gaussian proxy used for policy updates. We hope these explanations and results address the main questions about our framework, and welcome further discussion. #### **References** **[1]** Xiong, K., Guo, X., Li, F., Yan, S., Xu, G., Zhou, L., ... & Wang, X. (2026, April). Recogdrive: A reinforced cognitive framework for end-to-end autonomous driving. In International Conference on Learning Representations (Vol. 2026, pp. 157518-157556). **[2]** Zou, J., Chen, S., Liao, B., Zheng, Z., Song, Y., Zhang, L., ... & Wang, X. (2025). Diffusiondrivev2: Reinforcement learning-constrained truncated diffusion modeling in end-to-end autonomous driving. arXiv preprint arXiv:2512.07745. **[3]** Jing, C., Tian, H., Li, H., Li, T., Wang, C., Zhang, Z., ... & Yang, Z. (2026). SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models. arXiv preprint arXiv:2610.13781.
Reviewer_Gk9yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

DreamerAD adapts pretrained video world models to train autonomous-driving policies through reinforcement learning. It uses shortcut training for efficient trajectory-conditioned latent prediction, a reward model that predicts eight driving metrics at eight future horizons, and Gaussian-weighted sampling from a filtered trajectory vocabulary. The candidate rewards are aggregated and used in a clipped, group-relative Gaussian surrogate objective, together with behavioral cloning. The framework is implemented with Epona and a DriveVA-initialized Wan2.2 backbone.

The complete training procedure raises NAVSIM v2 EPDMS from 85.1 to 87.8 for Epona and from 89.1 to 91.4 for Wan2.2. Zero-shot HUGSIM HD-Score increases from 29.71 to 32.40 and from 46.68 to 54.96, respectively. The experiments also cover component ablations, rollout latency, reward-data scaling, and trajectory-ranking accuracy, with qualitative examples of both successes and failures.

Critical strengths
  1. Tables 1-3, on pages 7-8, show improvements over the corresponding backbones on both NAVSIM versions and zero-shot closed-loop HUGSIM. The 8.28-point HD-Score improvement for Wan2.2 on HUGSIM is particularly convincing because it shows benefits beyond the environment used to generate reward supervision.

Appendix D.2 breaks down performance by difficulty, and Appendix D.4 acknowledges the remaining failures in Extreme scenarios. These results support the practical contribution and give a clear account of its limitations.

  1. Table 5 directly measures the trade-off between rollout cost and downstream policy performance. Reducing the world-model sampling steps from 16 to one gives approximately 13.4× and 16× rollout speedups for Epona and Wan2.2, respectively. EPDMS stays at 87.8 for Epona and changes only from 91.5 to 91.4 for Wan2.2.

This connects faster generation to the quality of the trained policy. Table 4 also s

Critical weaknesses
  1. The experiments do not fully separate the contribution of latent imagination from additional supervision and direct trajectory scoring.

Section 4.4.1 describes the comparison between IDs 4 and 6 in Table 4 as isolating SF-WM. However, Section 3.2 and Appendix B.4/Table 6 state that shortcut training also updates the trajectory branch with ground-truth trajectory supervision. Appendix C.3 likewise retains trajectory supervision in the shared Wan2.2 backbone. It is unclear whether the control without shortcut training receives the same supervised-update budget.

AD-RM also takes the observed context and candidate trajectory as inputs, alongside the generated future latents. Table 4 lacks a matched reward model that uses only the context and candidate trajectory. The complete procedure works, but these experiments do not show how much of the improvement requires generating future visual states.

Since evaluation through imagined future states is central to the method, I would like to see intermediate policy checkpoints and a matched reward model without future latent inputs.

  1. The reward-model evaluation does not closely match the candidate distribution used during RL.

Appendix D.6/Table 10 reports strong pairwise ranking accuracy on randomly sampled non-expert vocabulary trajectories. During RL, however, the method compares Gaussian-weighted candidates near the current policy prediction, as described in Section 3.4.

Good ranking accuracy on broadly sampled trajectories does not establish that the model can reliably distinguish nearby alternatives. A ranking error is especially concerning when it favors an unsafe trajectory. Candidates collected from actual RL checkpoints would provide a more relevant test. Their simulator scores and the safety errors of the preferred candidate would show whether the reward model remains useful during optimization.

Evaluating AD-RM on training scenes is not itself a methodological flaw, since it is a training-time component. The downstream benchmark gains still support its usefulness. My concern is whether the reported reward evaluation reflects the distribution on which the model guides learning.

Questions for authors
  1. In Table 4, IDs 4 and 6, were the trajectory branch, shared representation, and supervised-update budget matched? Please report policy performance immediately after shortcut training and after RL, along with a matched continuation using behavioral cloning alone.

  2. With the same vocabulary, reward labels, optimization budget, and policy initialization, how does a reward model using only the observed context and candidate trajectory perform? This would test how much the generated future latents contribute.

  3. Could you extend Table 10 with candidates collected from actual RL checkpoints and report simulator-measured safety errors for the preferred candidate? Please also provide uncertainty estimates for the main improvements and the smaller differences in Table 4. A scene-disjoint reward evaluation would add useful evidence, althoug

4 comments
AuthorsOpenReview ↗

Experimental Controls and Reward-Model Reliability

We thank the reviewer for carefully assessing our experimental controls, the contribution of future imagination, and reward-model reliability. Your comments help us strengthen comparisons, evaluate reward prediction on candidates sampled using the RL procedure, and examine variability across training runs.

W1/Q1: Matched supervision and the BC-only control

Matched supervision and update budget. Thank you for highlighting the need to distinguish the effects of SF and RL from additional supervised training. We therefore make the supervision budgets explicit and include a BC-only continuation. IDs 4 and 6 in Table 4 both start from ID 1 and undergo 3 epochs of pre-RL training, updating the same modules with the same trajectory supervision. They differ only in the SF training objective, and both subsequently receive the same 2-epoch RL budget. Table G1 reports performance at each stage, including results for the BC-only control with a matched budget.

Table G1. Policy performance under matched training budgets. A dash (-) denotes not applicable.

VariantInitial EPDMSPost-control EPDMS (3 epochs)Post-SF EPDMS (3 epochs)Post-RL EPDMS (2 epochs)Post-BC-only EPDMS (2 epochs)
ID 4: matched control + RL89.189.3-90.5-
ID 6: SF + RL89.1-88.891.4-
SF + BC-only89.1-88.8-89.0

Performance immediately after SF. During SF training, trajectory performance initially decreases and then recovers, ending slightly below its initial level. This may reflect the trajectory branch adapting to changes in the shared representation.

RL versus additional BC updates. The RL continuation for ID 6 and the BC-only control start from the same post-SF model and form two separate training branches. Full RL achieves a final EPDMS 2.4 points higher than BC-only. Additional BC updates therefore do not explain the improvement; the main gain comes from RL optimization.

W1/Q2: Context-only RM control and the contribution of imagined futures

Reward prediction without future latents. We agree that the value of future imagination should be evaluated against a control using only the observed context and candidate trajectory. The RM with imagined future latents performs better in both reward prediction and downstream policy evaluation. We compare an RM using only the observed context and candidate trajectory with one that additionally uses future latents generated by conditioning on the candidate trajectory. Both RMs are trained for 2 epochs. Evaluation covers all 12,146 navtest scenes, with 8 candidates per scene: random-vocabulary candidates are randomly sampled from the candidate vocabulary, while RL candidates are generated using the RL sampling procedure from the reported policy checkpoint with 91.4 EPDMS.

MAE measures the mean absolute error in the aggregated log reward. PA is pairwise ranking accuracy averaged over valid scenes, excluding ground-truth ties and awarding half credit to predicted ties. The false-safe selection rate measures how often the trajectory with the highest RM-predicted aggregate reward is predicted safe but judged unsafe by the simulator: its predicted NC, DAC, and TTC at 4 s are all ≥0.5, while at least one ground-truth value is 0. The denominator is the total number of evaluated scenes.

Table G2. RM prediction performance on two candidate distributions.

RM inputRandom-vocabulary MAE ↓RL-candidate MAE ↓Random-vocabulary PA ↑RL-candidate PA ↑False-safe selection rate ↓
Context + trajectory0.720.3178.83%68.23%6.90%
Context + trajectory + imagined future0.480.2782.64%77.62%3.71%

Table G2 shows lower MAE and higher PA for the imagined-future RM on both candidate distributions. On RL candidates, PA increases by 9.39 percentage points, and the false-safe selection rate is also lower. These results support the value of future latents beyond the observed context and candidate trajectory when evaluating candidates near the policy prediction.

Downstream policy performance. The imagined-future variant also achieves higher final NAVSIM EPDMS and HUGSIM HD-Score. Table G3 reports the actual pre-RL and post-RL results for both branches.

Table G3. Policy performance before and after RL with different RMs.

RM inputPre-RL EPDMSPost-RL EPDMS ↑Pre-RL HD-ScorePost-RL HD-Score ↑
Context + trajectory88.891.046.1051.59
Context + trajectory + imagined future88.891.445.7954.96

Training progress. After 1 epoch, the imagined-future RM reaches 77.92% random-vocabulary PA, close to the context-only RM's 78.83% after 2 epochs. Under the same 2-epoch budget, the imagined-future RM reaches 82.64%. Future latents provide intermediate scene states conditioned on the candidate trajectory, which may ease reward learning at each horizon.

AuthorsOpenReview ↗

Reward-Model Reliability

W2/Q3: RM reliability and variability across runs

Reward accuracy and safety near the policy. Accuracy on broadly sampled vocabulary trajectories does not by itself establish reliability near the policy prediction. We therefore evaluate reward error, ranking accuracy, and false-safe selections on candidates generated using the RL sampling procedure. For each of the 12,146 navtest scenes, we generate 8 candidates from the reported policy checkpoint with 91.4 EPDMS.

MAE measures error in the aggregated log reward. PA is pairwise ranking accuracy averaged over valid scenes, excluding ground-truth ties and awarding half credit to predicted ties. The RM selects the trajectory with the highest predicted aggregate reward without first filtering candidates for safety. A false-safe selection occurs when the selected trajectory's predicted NC, DAC, and TTC at 4 s are all ≥0.5, but at least one simulator ground-truth value is 0. The rate uses the total number of evaluated scenes as its denominator.

Table G4. Reward prediction and false-safe selections on actual RL candidates.

RM inputRL-candidate MAE ↓RL-candidate PA ↑False-safe selection rate ↓
Context + trajectory0.3168.23%6.90%
Context + trajectory + imagined future0.2777.62%3.71%

As Table G4 shows, the imagined-future RM retains lower reward error, higher ranking accuracy, and a lower false-safe selection rate on candidates near the policy prediction. This complements the evaluation on broadly sampled vocabulary trajectories in Table 10.

Variability across training runs. Small differences between variants should be interpreted alongside variability across independent training runs. We therefore repeated training twice for each of IDs 2, 3, 4, and 5 using different random seeds.For each run, we selected the checkpoint with the highest navtest EPDMS. Table G5 lists the originally reported result, the two repeat runs, and the mean and sample standard deviation across all three results.

Table G5. Independent repeat runs for IDs 2–5 in Table 4.

Table 4 IDReported EPDMSRepeat 1Repeat 2MeanSample SD
289.589.389.689.470.15
390.189.689.989.870.25
490.590.490.590.470.06
590.890.990.990.870.06

ID 3 varies more across runs, while the other variants have means close to their originally reported scores. The three-run means retain the ordering ID 2 < ID 3 < ID 4 < ID 5, continuing to support our conclusions from these ablations.

Summary Response

The additional BC-only and context-only RM comparisons provide further evidence for the contributions of policy refinement and imagined future states. Our navtest evaluation examines reward prediction, candidate ranking, and false-safe selections on candidates sampled using the RL procedure. We also report repeated runs for IDs 2–5, whose mean scores retain the original ordering. We appreciate the reviewer’s suggestions for strengthening these controls and evaluations, and welcome further questions or suggestions.

Thank you for the detailed response. The new controls address my main concerns. In particular, the matched training budgets and BC-only continuation clarify that the improvement is not simply due to additional supervised updates. The context-and-trajectory-only RM is also a useful control. The evaluation on candidates from the actual RL policy and the repeated runs are also helpful. Overall, I consider my concerns substantially addressed and will raise my rating to weak acceptance. I encourage the authors to include these additional results in the final manuscript.

AuthorsOpenReview ↗

Thank You!

Thank you for the thoughtful feedback. Your suggestions helped us strengthen the experimental comparisons, and we’re glad the additional results addressed your main concerns. We also appreciate your willingness to raise your rating. We will include these results in the final manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.