acceptodds
Under review as a conference paper at ICLR 2027

Closing the Loop on Open-Loop Driving Data: Reinforcement Learning of Action Policies with Controllable World Models

Abstract

Policies trained from recorded driving logs rarely observe how their own decisions change future observations, limiting their ability to recover once they depart from the recorded trajectory. We present DriveVerse, an autoregressive multi-view world model that turns logged scenes into interactive environments for policy learning. At each decision step, a structured simulation backend executes part of the policy plan and constructs view-aligned map and motion conditions; DriveVerse then generates synchronized camera observations for the next decision. These post-action observations enable recurrent policy interaction beyond the recorded ego trajectory. Shared cross-view attention, causal training, and self-rollout distillation support recurrent generation, while online diagnostics monitor rollout reliability. We optimize the resulting rollouts with T-GRPO, combining plan-level and executed-trajectory feedback across vision-language-action, world-action, and end-to-end policies through their native action representations. Experiments show that DriveVerse outperforms the strongest baseline, lowering front-view FVD by 22% and improving all reported object- and lane-geometry metrics and temporal consistency. Interactive training with DriveVerse improves all three policy families (VLA, WAM, E2E) over their SFT and RL baselines on both navtest and navhard. The VLA reaches 92.8 and 52.3 EPDMS, respectively, improving on its GRPO baseline by 1.7 and 5.2 points.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, world model, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fwbeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DriveVerse, a controllable autoregressive multi-view observation generator for closed-loop driving-policy training. A structured simulation backend maintains the world state, and a world model generates synchronized multi-view future camera frames conditioned on surrounding-agent states, road geometry, and traffic signals. A four-stage training pipeline combines bidirectional adaptation, diffusion forcing, consistency distillation, and distribution matching on generated histories. Online diagnostics reject unreliable observations. Policies are optimized using an adopted T-GRPO objective that combines local plan rewards and executed-trajectory feedback. Experiments report improved video and geometry metrics and higher NAVSIM v2 EPDMS for VLA, WAM, and E2E policies. The VLA improves from 91.1/47.1 to 92.8/52.3 on navtest/navhard relative to its GRPO baseline.

Critical strengths
  • A useful integration of controllable generation and interactive policy learning.The backend explicitly owns scene dynamics and rewards, while the generator renders the camera views for planning. Perspective-map and BEV conditioning, causal generation, and training on generated histories address complementary obstacles to recurrent interaction. Tables 2 and 3 support these design choices on generation metrics. This is a consequential systems contribution toward reusing logged scenes for visual policy training.

  • Consistent policy improvements across three action representations. Table 4 reports gains over both SFT and the selected RL baseline for each policy family on both benchmarks. Table 5 also matches VLA initialization, scenes, sample counts, and update budgets, and shows that combining step and trajectory advantages outperforms either alone. These results make the full pipeline promising beyond a single policy architecture.

Critical weaknesses
  • The policy experiments do not sufficiently establish which part of the pipeline produces the gains. Table 5 provides useful controls over initialization, scenes, sample counts, and update budgets, and the online step-only variant already improves over logged-observation GRPO. However, it remains unclear whether these two settings use the same EC-aware training reward. Clarifying this would strengthen the attribution of the gains to post-action observation feedback. More broadly, the generator ablations in Tables 2–3 are evaluated only on generation metrics, leaving their contribution to downstream policy improvement unestablished.
  • The reliability of generated observations after the policy deviates from the recorded trajectory is not sufficiently validated. The quantitative generation evaluation uses recorded ego motion, whereas the recurrent reliability result is acceptance by the proposed diagnostics on a prefiltered subset: 65.3% of test clips are retained, and 87.2% of attempted rollouts within this pool complete four seconds. These figures transparently describe conditional coverage, but do not quantify the rate of materially incorrect observations among accepted segments. Manual inspection near the validation threshold is useful calibration, yet no quantitative independent audit or evaluation of reliability under the final RL policies is reported. NAVSIM's original and pre-generated follow-up observations provide useful policy evaluation, but do not independently validate the generator during recurrent interaction. This limits confidence that the diagnostics protect learning when optimized policies visit states different from those of the frozen SFT policy used for the reliability experiment.
Questions for authors
  • Do the logged-observation GRPO baseline and the online step-only variant in Table 5 use the same EC-aware training reward and scoring implementation? If they differ, could you clarify these differences and their potential contribution to the gains? Additionally, do you have a controlled policy-training comparison using one of the generator ablations in Tables 2–3, with the remaining training settings fixed, to establish whether improved generation quality translates into better policy performance?
  • Beyond the diagnostics’ own acceptance rates, have you independently assessed accepted generated observations for object or lane errors that could affect driving decisions? If so, could you report the assessment procedure and error rates? Since the reported recurrent reliability experiment uses a frozen VLA SFT policy, could you also provide acceptance rates and corresponding observation-quality assessments for rollouts collected with the final RL policies? This would help establish whether the diagnostics remain effective under the state distributions encountered after policy optimization.
2 comments

W1 and Q1: Training Reward Formulation and Generator Contribution to Policy Learning

We thank the reviewer for the constructive assessment and questions. Below we explain how the training reward operates across single-step and online settings and how world-model generation quality and distillation govern policy learning.

Both settings use the same NAVSIM v2 EPDMS scorer, and extended comfort is the one term GRPO on logged observations has no second plan to compute, as in prior work. Using the benchmark's sub-scores, thresholds, and weights, NAVSIM v2 EPDMS multiplies compliance terms by a weighted average of five terms: progress, time-to-collision, lane keeping, history comfort, and extended comfort. Four of these five score one plan alone, whereas EC compares the plan at one observation with the plan at the next (Appendix L1007). Because logged-observation GRPO draws unexecuted candidate plans at one decision step, no post-action observation or second plan exists, and the baseline uses the standard reward of this line of work, EPDMS over the four defined terms with benchmark weights. Every DriveVerse rollout scores its first decision step without EC like the baseline (Appendix L1014). From the second decision step onward, the policy replans from the generated post-action observation, so both plans exist and EC is scored with its benchmark weight. The step-only variant thus differs from the baseline only in the availability of post-action observations, and EC supervision on states reached by the policy's actions is part of what this feedback contributes (L436, Appendix L1012).

SettingDecision stepWeighted terms in the step reward
Logged-observation GRPOSingleProgress, time-to-collision, lane keeping, history comfort
DriveVerse step-onlyFirstProgress, time-to-collision, lane keeping, history comfort
DriveVerse step-onlySecond to eighthThe four terms above and EC

Leaving EC out on a single logged scene is established practice, and the gains appear in terms that both rewards contain. FlowR2A [1] excludes the two-frame EC term because it is not defined within a single scene, and GRPO-based planners such as WAM-Flow [2] and ExploreVLA [3] likewise optimize rewards without EC and report NAVSIM v2 EPDMS. Under matched VLA initialization, scenes, sample counts, and update budgets (L429), navtest / navhard EPDMS rises from 91.1 / 47.1 for logged-observation GRPO to 92.1 / 49.4 for DriveVerse step advantage only, 91.8 / 50.0 for trajectory advantage only, and 92.8 / 52.3 for T-GRPO (L415). For the VLA on navtest, T-GRPO relative to GRPO raises ego progress from 90.0 to 93.4 and improves collision avoidance and drivable-area compliance, while EC rises from 82.0 to 86.0 (L423). On navhard, first-stage and second-stage scores rise from 82.0 to 87.6 and from 56.8 to 59.6 (L427). Progress and compliance terms are scored identically in the baseline, so these gains arise in quantities the baseline reward already optimizes.

Generator quality reaches the policy through the reliability gate, while causal distillation makes interactive rollouts computationally practical. A generated segment is returned to the policy only if it passes our diagnostics; a rollout rejected at its first segment supplies no post-action observation and reduces to a logged-observation GRPO sample, moving training toward the logged-observation GRPO row of Table 5. Under the same diagnostics on our held-out test clips, the strongest baseline world model, Cosmos 3 Nano, reaches an acceptance rate of only 22.7%, against 65.3% for DriveVerse (L407), as slight shifts in perspective cause serious lane-topology mismatch and other road users move without control. Finally, causal distillation reduces denoising from 35 steps per chunk in the bidirectional teacher to four (Table 3, L340, L397), avoiding the approximately 8 times longer rollout time of the undistilled bidirectional model, which would turn RL training from days into weeks.

We hope these responses have brought added clarity to our work, and we welcome further comments achievable within the coming week before the discussion deadline if that would help your assessment.

[1] X. Li, Z. Liu, X. Ye, et al. FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning. arXiv:2606.24231, 2026.

[2] Y. Xu, J. Cui, F. Cai, et al. WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving. CVPR, 2026.

[3] Z. Sheng, X. Ye, J. Luo, et al. ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving. arXiv:2604.02714, 2026.

W2 and Q2: Observation Quality and Diagnostics Under Policy Optimization

We thank the reviewer for asking how generated observation quality is verified off the logged trajectory and under policy optimization.

Object and lane errors of the generated observations are measured independently of the online gate. On 1,000 navtest clips under recorded ego motion in Table 1 (L324), frozen BEVFormer for 3D detection and frozen LATR for lane detection show DriveVerse reaching LET-AP 0.407, LET-APL 0.261, LATR-F1 0.892, and a far-range lateral lane error of 0.306, the best values among all models compared in Table 1. Neither BEVFormer nor LATR is part of the online gate, which instead uses YOLOv12 for objects and CLRerNet for lanes (L248, Appendix L802), so the two sets of perception models check each other.

Once the policy leaves the logged trajectory, every generated segment is checked at the pose the policy actually reached. Because no recorded image exists at the new poses, metrics that need a reference image do not apply, and the established practice for such novel trajectories is to run perception models on the generated frames and compare the detections with the scene annotations, for which DriveDreamer4D [1] introduces NTA-IoU and NTL-IoU. Following the same design, our online object and lane agreement scores compare detections on every generated segment with the agents and lanes projected from the simulated state (Appendix L812), with tolerances that accommodate detector and projection error (Appendix L821). The acceptance threshold was calibrated on motion that leaves the log across 1,771 validation clips with 11 trajectory variants built following the navhard construction, with manual inspection of clips near the boundary (Appendix L842), and a segment is accepted only if the minimum of all its scores passes (L255).

The policy results are the direct test of whether any error left in accepted observations affects driving. DriveVerse is used only to train, and every policy is evaluated with the official NAVSIM v2 evaluation on the observations the benchmark provides (L360), with no DriveVerse-generated observation entering evaluation. If accepted segments placed objects or lanes wrongly in ways that affect driving, a policy trained on them would learn responses that do not hold on the benchmark's observations, and its benchmark score would not improve. What is observed is the opposite, as all three policy families improve over their RL baselines on both navtest and navhard in Table 4 (L378), and on the second stage of navhard, which evaluates the policy from follow-up states off the logged trajectory, EPDMS rises from 56.8 to 59.6 for the VLA, from 41.0 to 52.8 for the WAM, and from 46.9 to 54.2 for the E2E policy relative to their RL baselines (L440). The navtest EPDMS gains also hold on both sides of the initial reliability filter, which accepts 653 and rejects 347 of the 1,000 clips of Table 1:

Policy (baseline RL -> DriveVerse T-GRPO)Accepted scenesRejected scenes
VLA (GRPO)94.4 -> 95.691.3 -> 93.0
WAM (FlowGRPO)87.8 -> 95.182.6 -> 92.2
E2E (WorldEngine RL)90.1 -> 94.685.4 -> 91.6

Every family improves on both the accepted and the rejected subsets.

The gate is applied throughout training whichever policy acts, so it covers the states that optimized policies visit. The diagnostics are not tied to the SFT policy; they run online on every generated segment of every training rollout, and the object and lane checks compare the segment with the simulated state that the current policy's own action produced before the segment reaches the policy. A state far from real driving data lowers the distributional support score (Appendix L824), and the segment is rejected with no reward penalty once the reliability score falls below the threshold, while the rollout is still scored by the simulator over the full 4 s horizon (L305), so an optimized policy that moves outside the reliable range of the world model receives a simulator score, not an unreliable image. The reported experiment uses the frozen VLA SFT policy that T-GRPO training starts from, with fixed checkpoints and diagnostic parameters on reserved test clips, to give one reproducible reference figure (L405, Appendix L856). The final policies are the product of this training and are evaluated on the official benchmark, where the world model plays no part: the world model's task ends with training.

We hope these responses have brought added clarity to our work, and we welcome further comments achievable within the coming week before the discussion deadline if that would help your assessment.

[1] G. Zhao, C. Ni, X. Wang, et al. DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation. CVPR, 2025.

Reviewer_K5FiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a closed-loop driving simulation stack based on a controllable multi-view world model. It adopts online sample diagnostics and the proposed T-GRPO to train heterogeneous driving policies including VLA, WAM and E2E. It alleviates the open-loop bottleneck of log-based training and obtains steady improvements on NAVSIM v2. A main limitation is its reliance on HD maps and scene annotations.

Critical strengths
  1. This closed‑loop simulation pipeline is a key strength. Interactive driving world modeling is a challenging frontier problem, and this work delivers a fully validated stack combining controllable multi‑view generation, online reliability diagnostics and T‑GRPO reinforcement learning. It mitigates the open‑loop limitation of log‑based training and yields clear policy improvements on NAVSIM v2

  2. The RL design is a notable strength. T‑GRPO unifies optimization for heterogeneous VLA, WAM and E2E policies using both step‑wise and trajectory‑level advantages. Closed‑loop training is stabilized by online diagnostics, with consistent gains over SFT and RL baselines on NAVSIM v2

Critical weaknesses

1.The paper aims to provide a reliable interactive environment for policy learning, but its reliability evaluation remains limited in scope. The reported 87.2% rollout acceptance rate covers only 4-second interactions in the 65.3% of test scenes retained after initial filtering, while the 20-second experiments follow recorded trajectories. The paper neither characterizes the excluded scenes nor evaluates longer policy-driven interactions. Reliability in challenging scenarios and sustained interaction therefore remains unclear, limiting the evidence for the framework’s broader applicability

Questions for authors

When the ego vehicle briefly reverses or revisits a previously observed area, can the model preserve scene appearance consistent with earlier observations, particularly details not encoded in the layout? Can the current diagnostics, which primarily assess visual quality and object and lane agreement, detect such inconsistencies across revisits?

2 comments

W1: Scope of the Reliability Evaluation

We thank the reviewer for raising questions regarding the filtered scenes, performance in challenging scenarios, and longer-horizon policy rollouts. Below, we clarify how the reliability gate operates during training and evaluation, and we provide new breakdowns across excluded scenes, policy performance on accepted versus rejected subsets, and policy-driven interactions up to 20 s. We hope these additional results bring more clarity, and we will include the four tables in the appendix of the revised paper.

First, the initial reliability filter only selects which scenes serve as the starting point for training-time interaction; it is never applied at evaluation. Every policy result in Table 4 uses the official NAVSIM v2 evaluation on the full navtest and navhard splits. During training the initial step rewards are always kept, while the remainder of the 4 s scoring horizon is completed using the latest plan (Appendix L1080). Even if the very first generated segment is rejected, the sample falls back to a standard logged-observation GRPO sample. Low-reliability scenes therefore still contribute to training, while accepted multi-step rollouts substantially enrich the optimization signal. Under logged-observation GRPO, 42.1% of scene groups receive all 0 or 1 rewards within group and yield zero advantage, whereas with T-GRPO in DriveVerse, 41.6% of groups have zero advantage at the first step and only 7.4% remain at zero advantage over 8 steps.

Stationary and low-speed scenes are mostly retained, while rejections concentrate in high-speed, mostly straight driving, followed by intersections and multi-command segments. To characterize which scenes the initial filter excludes, we break down the 1,000 navtest clips from Table 1 by nuPlan scenario tag and navigation command:

nuPlan scenario tagAcceptedRejected
high_magnitude_speed69145
traversing_traffic_light_intersection107100
traversing_intersection146171
stationary40122
stationary_at_traffic_light_without_lead603
low_magnitude_speed483
Navigation commandAcceptedRejected
straight439223
left497
right572
unknown228
two or more86107
Total653347

(A) On high-speed driving scenarios, the rejection mainly happens due to the blurring cause by fast motion, which is a broadly recognized challenge for video world models [1,2]. (B) On the intersections, it usually features denser traffic and more complex road structure. We consider them as more challenging scenarios regarding object and lane structure controllability. This aligns with our Limitations discussion noting that controllability is bounded by the scenes and motion the world model can faithfully reproduce.

Crucially, excluding these scenes from multi-step training rollouts does not restrict policy gains to the accepted subset. Evaluating navtest EPDMS across the same 1,000 clips split by initial filter acceptance shows consistent improvements across both subsets:

Policy (baseline RL -> DriveVerse T-GRPO)Accepted scenesRejected scenes
VLA (GRPO)94.4 -> 95.691.3 -> 93.0
WAM (FlowGRPO)87.8 -> 95.182.6 -> 92.2
E2E (WorldEngine RL)90.1 -> 94.685.4 -> 91.6

The rejected scenes are indeed more challenging for all baselines (lower baseline EPDMS), yet DriveVerse T-GRPO improves every policy family on the rejected subset as well.

Finally, rollout quality changes slowly and steadily when the policy-driven horizon is extended from 4 s to 20 s. While policy training uses 4 s rollouts to match the NAVSIM scoring horizon, we also evaluated longer policy-driven interactions using the frozen VLA policy on the retained test clips, replanning every 0.5 s with the reliability check applied at every segment:

Horizon4 s8 s12 s16 s20 s
Full-rollout acceptance87%73%60%49%39%
Per-step acceptance98.3%98.1%97.9%97.8%97.7%

Because full-rollout acceptance is a cumulative criterion where a single rejected segment fails the entire rollout and remains in the denominator, the overall rate naturally decreases as the horizon grows even if single-step generation quality stays constant. The calculated per step acceptance row isolates step-level quality from this compounding effect, which moves gradually down, showing a slow, steady change in rollout quality during sustained policy-driven interaction.

[1] T. Xia, Y. Li, L. Zhou, et al. DriveLaW: Unifying Planning and Video Generation in a Latent Driving World. CVPR, 2026.

[2] W. Lee, K. Kotar, R. M. Venkatesh, et al. Unified 3D Scene Understanding Through Physical World Modeling. ICLR, 2026.

Q1: Scene Consistency When the Ego Vehicle Reverses or Revisits an Area

We thank the reviewer for this thoughtful question on how DriveVerse handles spatial revisits.

Returning to an earlier spot almost never happens in nuPlan/NAVSIM's driving data, so it lies outside the regime in which our policy is trained and evaluated. Across the navtrain split, excluding stationary clips, only 20 of the 103,288 clips involve the ego vehicle returning to an earlier spot, with an average ego speed of 1.2 m/s.

When an area does stay in view or come back into view during a rollout, DriveVerse preserves the properties that govern driving decisions by construction. At every step, generation is conditioned on the structured world state projected into each camera view (L142), so object locations and lane geometry come directly from the simulation rather than visual memory. The global look of the scene remains tied to the initial real observation through the pinned first latent and the sliding window of recent latents (Appendix L787). Fine textures not encoded in the layout may be re-synthesized with some variation, which acts as benign visual augmentation for policy learning as long as layout geometry and global appearance are preserved.

Our online diagnostics follow the same scope because of how the policy consumes observations. Every evaluated policy observes at most the past 2 s during the 4 s training rollouts, so a view from 20 s earlier never enters the policy inputs or changes its decisions. Within that window, each generated segment is verified for object and lane agreement with the simulated state, perceptual drift from the initial context frame, and frame-to-frame temporal consistency before the rollout continues (L248, L255).

We hope these responses have brought added clarity to our work, and we welcome further comments achievable within the coming week before the discussion deadline if that would help your assessment.

Reviewer_CLnDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DriveVerse, an autoregressive, controllable, multi-view world model for converting logged driving data into an interactive environment for policy optimization. The central motivation is that policies trained on open-loop driving logs do not observe the consequences of their own actions and therefore suffer from covariate shift and weak recovery behavior.

DriveVerse couples three components:

A deterministic structured simulator that updates ego and traffic states after executing a short segment of the policy plan. A controllable multi-view video generator that synthesizes camera observations conditioned on the updated scene state, ego motion, perspective map rasters, and bird’s-eye-view scene tokens. An online policy-optimization loop based on T-GRPO, which combines step-level planning rewards and trajectory-level driving rewards. The world model is adapted from Cosmos 3 and trained with bidirectional teacher fine-tuning, chunk-causal diffusion forcing, consistency distillation, and long-horizon distribution-matching distillation. The authors additionally introduce online rollout diagnostics based on video quality, object/lane agreement, and distributional support; generated observations are fed back to the policy only when they pass a reliability threshold.

Critical strengths
  1. The paper tackles a central weakness of open-loop driving policy training: a policy can only be trained on states present in the recorded trajectory, even though its own actions may lead to different future states. The proposed interaction loop is well motivated and directly targets this limitation.

The paper correctly recognizes that a useful driving world model must generate more than visually plausible frames. It must preserve:

ego-motion consistency, multi-view synchronization, road and lane geometry, object placement, temporal consistency under repeated replanning. This distinction between visual realism and controllable, policy-useful simulation is important.

  1. The combination of perspective map residuals, BEV tokens, and ego-motion conditioning is technically sensible. The perspective map provides dense camera-aligned structure, while BEV tokens preserve geometry that may be outside the camera frustum. This is especially relevant to the failure case illustrated in Figure 1, where an initially occluded side street is lost when the model relies only on image history and perspective conditioning.

The use of shared cross-view attention is also well motivated for synchronized multi-camera generation. The ablation results suggest that shared attention improves FVD and lane agreement relative to a dedicated cross-view attention design.

  1. A major positive aspect is that the paper does not evaluate the world model only under teacher-forced or recorded-motion conditions. It explicitly considers recurrent generation, where model-generated frames are fed back as future policy observations.

The proposed rollout diagnostics—video quality, object/lane agreement, and distributional support—are useful engineering safeguards. The authors also report recurrent acceptance rates, rather than hiding the fraction of generated rollouts that fail the reliability criteria.

  1. Applying the same interaction framework to VLA, WAM, and E2E policies is valuable. The policies use different native action representations:

VLA: reasoning and trajectory-token likelihoods, WAM: transition likelihoods through a trajectory-denoising chain, E2E: categorical likelihoods over trajectory candidates. This demonstrates that the proposed environment is not limited to a single policy parameterization.

  1. The quantitative results are promising:

DriveVerse reduces front-view FVD from 22.7 to 17.7 relative to OmniDreams. It improves object and lane feature agreement metrics. DriveVerse T-GRPO improves the VLA from 91.1 to 92.8 on navtest and from 47.1 to 52.3 on navhard relative to the reported GRPO baseline. The WAM improves from 90.3 to 92.0 on navtest and from 37.3 to 44.8 on navhard. The E2E policy improves from 88.8 to 91.4 on navtest and from 32.7 to 43.6 on navhard. The gains are particularly notable on navhard, where post-action observations and trajectory-level rewards should matter most.

  1. The paper includes useful ablations for:

map residuals, BEV tokens, ego-motion tokens, shared versus dedicated cross-view attention, bidirectional teacher fine-tuning, causal diffusion forcing, consistency distillation, long-horizon distribution matching distillation, step-level versus trajectory-level advantages. These experiments support the claim that the final system is not simply the result of increasing model size or adding a generic RL objective.

  1. The paper acknowledges that world-model controllability is bounded by the scenes and motion conditions it can reliably reproduce. This is an important limitation and should be emphasized because it directly affects how broadly the method can be used for policy improvement.
Critical weaknesses
  1. The initial reliability filter retains only 65.3% of the held-out test clips and 71.3% of the training clips. Policy interaction is then conducted within this retained pool, and approximately 87.2% of attempted rollouts from the retained test clips complete all four seconds.

This creates an important ambiguity: are the reported policy results measured over the full NAVSIM distribution, or only over scenes that the world model can reproduce reliably?

If evaluation is restricted to the retained clips, then the policy results may partly reflect a favorable subset selection. This is especially important because the rejected clips could contain exactly the challenging or long-tail scenes where interactive policy learning would be most valuable.

The paper should report:

results on the full evaluation set, results on the retained subset, the fraction of clips retained for each split, performance stratified by reliability score, policy performance on clips where DriveVerse rejects interaction, a comparison against a policy trained/evaluated without initial filtering. The current presentation reports acceptance rates but does not fully quantify the effect of filtering on the final driving scores.

  1. When a generated observation fails the diagnostic threshold, the rollout terminates before the next policy decision. The executed prefix is retained, but the paper states that “the diagnostic stop itself carries no penalty”; the remaining horizon is completed using the latest full plan in the structured backend.

This design may make diagnostic rejection artificially benign. A policy can receive credit for the successfully executed prefix while avoiding subsequent interaction in cases where its observations are unreliable. More importantly, the policy is not directly incentivized to produce states that remain within the world model’s reliable support.

The authors should provide ablations for at least:

rejection with no penalty, as currently used; a fixed penalty for rejection; rejection treated as a failure; continuation using an alternative observation source; rejection excluded from training but counted as failure at evaluation. It would also be useful to report how often rejection occurs for each policy and at which decision horizon. Without this analysis, the final policy improvements may partly result from selectively truncating difficult rollouts.

  1. The policy receives generated camera observations, but driving rewards are computed from the structured simulator state and PDM-style scoring. The structured state is more accurate than the generated images and remains hidden from the policy, which is reasonable for scoring. However, this creates a potential mismatch:

the policy is optimized using rewards from the true structured state; the policy makes decisions based on generated images that may contain errors; the world model is not updated during policy optimization. Thus, the method is not quite equivalent to policy learning in a learned environment. It is closer to policy optimization with generated observations and an external privileged reward backend.

This is not necessarily a flaw, but the paper should more clearly define what is being claimed. The authors should also include an ablation comparing:

generated observations with structured-state rewards, generated observations with rewards estimated only from policy-visible observations, recorded/ground-truth observations with the same reward, generated observations with and without the structured simulator’s future-state information during scoring. This would establish whether the gains come primarily from post-action visual feedback or from privileged reward computation.

  1. The world model is evaluated mostly under recorded ego motion, while policy improvement uses policy-induced motion Table 1 evaluates generation under recorded motion, while policy optimization uses actions generated by the current policy. These are different regimes. The latter is the regime that matters most for the paper’s central claim.

The paper does report recurrent rollout acceptance rates, but the main generation-quality metrics do not appear to evaluate the world model under the distribution of policy-induced actions. Since the policy is optimized using the generated observations, the relevant question is not only whether DriveVerse reconstructs logged futures, but whether it remains reliable after the policy deviates from the logged trajectory.

The authors should report generation quality under:

expert/recorded actions, SFT policy actions, baseline RL policy actions, DriveVerse-RL policy actions, increasingly perturbed or off-distribution actions. Metrics should include feature agreement, temporal consistency, rollout survival, and state/action distribution shift. This would test whether the world model remains useful exactly where closed-loop training is expected to provide an advantage.

  1. The paper compares DriveVerse T-GRPO to several released or open-source baselines, including NVIDIA’s GRPO recipe, SimWAM FlowGRPO, and WorldEngine’s HydraMDP RL checkpoint. These baselines differ in:

model architecture, action representation, training recipe, data usage, RL objective, number of rollout steps, simulator or observation source, initialization and parameter updates. The comparison is useful as a practical benchmark, but it does not isolate the contribution of DriveVerse from the contribution of the adopted T-GRPO implementation, reward design, or additional training budget.

The paper partially addresses this with Table 5 for Alpamayo-R1, but the following controls would strengthen the claims:

logged-observation T-GRPO using the same code and reward implementation; DriveVerse interaction without trajectory-level advantage; DriveVerse interaction with the same number of policy updates as the baseline; a replayed ground-truth-observation interaction condition; the same T-GRPO procedure applied to a weaker or independently trained video world model. The paper should also report total compute, number of policy rollouts, number of world-model calls, and training wall-clock time for each comparison.

  1. The paper presents both a new interactive world model and an online policy-optimization method using step and trajectory advantages. Table 5 shows that both advantage types help, but it does not fully separate:

DriveVerse as an observation generator, T-GRPO as an optimization algorithm, the trajectory-level reward, the diagnostic gating mechanism.

  1. The policy replans every 0.5 seconds for up to eight decisions, corresponding to a four-second rollout. The paper motivates the method using delayed outcomes such as route completion and infractions, but the policy optimization horizon remains relatively short.

The trajectory-level reward includes closed-loop quantities, but the actual generated-observation interaction is only four seconds. The navhard evaluation includes a second-stage follow-up, yet the training protocol does not clearly indicate that the policy is optimized over comparable long-horizon interaction.

The authors should analyze performance as a function of interaction horizon, for example 2 s, 4 s, 8 s, and 16 s, and report whether gains persist or degrade as recurrent generation continues. This is especially important because the method explicitly claims to address long-horizon policy interaction.

8.The online threshold τ=0.74 is selected using manual inspection of generated clips near candidate thresholds. While this is understandable for a difficult visual-quality problem, it introduces a degree of subjectivity and may complicate reproducibility.

The paper should provide:

the candidate thresholds considered, the number of manually inspected clips, the annotator protocol, inter-annotator agreement, if multiple annotators were used, a quantitative criterion for selecting the threshold, sensitivity curves for τ. It would also be useful to report final policy performance across a range of thresholds rather than only at the chosen value.

  1. The reliability score may be conservative but not necessarily sufficient The gating score is defined as the minimum of video, object, lane, and support scores. This makes acceptance require all components to pass. However, the diagnostics depend on pretrained detectors and feature extractors, such as YOLOv12, CLRerNet, BEVFormer, and Cosmos Embed. Errors in these systems may either reject useful observations or accept visually plausible but semantically incorrect frames.

The paper should report diagnostic precision and recall with respect to actual rollout correctness. In particular:

How often does the diagnostic accept an incorrect observation? How often does it reject a correct observation? Which diagnostic component is most frequently responsible for rejection? Are object and lane detectors reliable under weather, occlusion, and unusual viewpoints? Does the support score favor conservative behavior close to the logged data and suppress useful exploration? The current diagnostics are plausible, but their relationship to policy improvement is not fully established.

  1. Some claims about “interaction” should be stated more precisely The structured simulator performs the actual state transition, while DriveVerse generates the visual observation returned to the policy. The world model therefore provides a learned observation transition conditioned on a structured transition backend; it does not model all environment dynamics itself.

This hybrid design is reasonable and likely beneficial. However, the paper should avoid implying that DriveVerse alone constitutes a complete world model of the driving environment. A more precise description would be a controllable learned observation model coupled to a structured scene-transition simulator.

This distinction matters for interpreting:

the model’s generalization, the source of the rewards, the meaning of “interactive world model,” the applicability to settings without structured annotations or a deterministic backend.

Questions for authors

The paper states that structured future states used to score candidate plans remain internal to the simulator. It would help to explicitly distinguish information available to the policy, the world model, and the reward function at every stage.

The VLA, WAM, and E2E policies use different trainable subsets and reference regularization. A concise table listing trainable modules, frozen modules, parameter counts, and learning rates would improve reproducibility.

The reported scores appear to be single values. Given the stochastic rollout sampling with group size N=8, results should include multiple random seeds or confidence intervals.

The T-GRPO procedure uses eight rollouts per initial scene. A sensitivity analysis for group sizes 4, 8, and 16 would clarify the stability of the method.

The local reward uses a weighted product of NAVSIM subscores, while the trajectory-level reward includes completion, infractions, and driving quality. The rationale for the weights and the re-estimation of λ should be explained more clearly.

Some baselines support different conditioning modalities and camera configurations. The table should clearly separate apples-to-apples comparisons from broader reference comparisons.

Four-step generation is an important efficiency claim, but the paper should report throughput, latency per generated segment, memory usage, and the cost of running diagnostics.

The language-understanding expert is frozen during world-model training, but the precise language inputs used by DriveVerse should be described. The fixed caption appears to be used for all samples; the motivation and effect of this design are unclear.

“Front-view FVD” and multi-view FVD should be defined carefully, especially because the model reports different values for front, left, and right views.

The paper would benefit from visual examples of accepted but incorrect observations, rejected observations, traffic-light errors, object identity errors, and lane-topology failures.

Are the final NAVSIM policy metrics computed on all evaluation clips or only on clips retained by the initial reliability filter?

What happens if diagnostic rejection is treated as a failure or given a negative reward? How much does the final EPDMS change?

How many policy rollouts are rejected during training, and how does the rejection rate evolve over training?

Does the world model generate observations under policy-induced actions with the same fidelity as under recorded actions?

Can the authors provide a controlled comparison using the same policy, reward, update budget, and T-GRPO implementation with and without DriveVerse interaction?

How much of the improvement comes from trajectory-level rewards versus post-action visual observations?

How is the policy behavior affected by changing the reliability threshold τ τ?

Does the support diagnostic discourage exploration away from the logged-data distribution?

What is the longest recurrent rollout tested under policy-induced actions, and how does reliability degrade with horizon?

Are the world model and diagnostic modules frozen throughout policy training? If so, how robust is policy optimization to systematic world-model bias?

How are moving-agent identities maintained across generated frames and structured annotations?

What is the computational cost of one 0.5-second world-model segment relative to policy inference and structured reward computation?

7 comments

W1, W2, and Q11–Q13: Evaluation Scope, Reliability Filtering, and Diagnostic Rejection

We thank the reviewer for the detailed review. Our response is organized into seven comments covering, in order: evaluation scope and diagnostic rejection in this comment; generation under policy-induced actions and interaction horizon; the roles of the simulator, the world model, and the reward; controlled comparisons; the diagnostics; and implementation details.

All reported policy results are evaluated on the full official NAVSIM v2 benchmark splits without any filtering. The initial reliability filter at threshold 0.80 under recorded ego motion (L256) and the online diagnostic gate exist only in training-time data collection, where the initial filter retains 71.3% of navtrain clips and 65.3% of held-out test clips for policy interaction (Appendix L835). DriveVerse is not part of evaluation at all: every number in the main comparison comes from the official NAVSIM v2 evaluation (L360) on the complete navtest and navhard splits with the observations the benchmark provides (Table 4, L378). Consequently, no subset selection can enter the reported scores, and all policies, ours and the baselines, are evaluated on the same complete splits. The initial filter applies only to DriveVerse T-GRPO training, because only that training generates observations with the world model; the GRPO, FlowGRPO and WorldEngine RL baselines involve no DriveVerse world model, so they need no such filter and are trained without it. The baseline rows therefore already are the policies trained and evaluated without initial filtering that the reviewer asks to compare against.

The policy gains also hold on the scenes that the initial reliability filter rejects. To verify this directly, we split the 1,000 navtest clips of Table 1 by the outcome of the initial filter into 653 accepted and 347 rejected clips, and compare the EPDMS of each policy family's baseline RL against DriveVerse T-GRPO; we will include this table in the appendix.

Policy (baseline RL -> DriveVerse T-GRPO)Accepted scenesRejected scenes
VLA (GRPO)94.4 -> 95.691.3 -> 93.0
WAM (FlowGRPO)87.8 -> 95.182.6 -> 92.2
E2E (WorldEngine RL)90.1 -> 94.685.4 -> 91.6

The rejected scenes are harder for every baseline policy as reflected by their lower baseline EPDMS, yet DriveVerse T-GRPO improves every policy family on them as well. Which scene types are rejected, mainly high-speed driving and intersections, is broken down by nuPlan scenario tag in our response to Reviewer K5Fi on W1.

A diagnostic stop cannot hide a difficult rollout because every rollout is still scored over the full 4 s horizon. When the online gate stops generation, the executed prefix and its step rewards are kept, and the latest full 4 s plan sampled before the rejection completes the remaining horizon in the structured backend (L305). Driving failures never terminate collection (L305), all incurred infractions remain, and every score component uses the same 4 s horizon (Appendix L1080). A plan that would crash is therefore penalized by the scoring backend whether or not its later observations were generated.

A diagnostic stop carries no reward penalty because it reflects the reliability range of the world model rather than the driving. A rejection reports that the observation generator left its reliable range (L305). Because this is a property of the world model and not of the driving, the paper treats it as neither a driving infraction nor a stall (Appendix L1091), whereas charging it to the policy would mix world-model reliability into a driving reward. The role of the stop is instead to bound how far a rollout relies on generated observations, which MBPO [1] does by construction when it trains on short model-generated rollouts branched from real data because errors compound over long model rollouts. Since the simulator still scores the remainder of the horizon, the stop leaves no part of a rollout unscored.

Online diagnostic rejection occurs on only a small fraction of attempted rollouts during training. Counting every sampled rollout in the denominator including stopped ones, 90.3% of attempted rollouts from filtered navtrain clips and approximately 87.2% from filtered test clips pass every one of the eight segments (Appendix L850). If the very first segment is rejected, the sample reduces to a standard logged-observation GRPO sample, so those scenes still contribute to training.

[1] M. Janner, J. Fu, M. Zhang, et al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS, 2019.

W4, W7, Q14, and Q19: Generation under Policy-Induced Actions and Interaction Horizon

We thank the reviewer for these questions on world-model evaluation and rollout length, and in this comment we address generation under the policy's own actions and the interaction horizon.

The paper evaluates DriveVerse directly under policy-induced actions in closed-loop rollouts, not only under recorded ego motion. With the frozen VLA SFT policy acting and replanning every 0.5 s, eight rollouts are sampled per retained test clip, and each generated 0.5 s segment becomes the visual history for the next policy decision and the next generation step (L405). Under these policy-induced actions, approximately 87.2% of all attempted rollouts keep every one of their eight segments above the 0.74 reliability threshold over the full 4 s horizon, and 90.3% complete on navtrain (Appendix L850). In addition, the online threshold itself was calibrated on off-log motion on the 1,771-clip validation split using 11 trajectory variants built following the navhard construction (Appendix L842).

Each motion regime is measured with the evaluation metrics that are defined for it. Under recorded ego motion, a held-out real future exists for 6.4 s continuations (L349), so we report FVD, PSNR, and Sampson error against those ground-truth frames (Table 1, L324). Once the policy leaves the logged trajectory, no recorded image exists at the new poses, so paired reference metrics such as PSNR are inapplicable and evaluation uses diagnostics that need no ground-truth images at counterfactual poses (L244). An established practice for such novel trajectories is to run off-the-shelf perception models on the synthesized frames and score them against the scene annotations. DriveDreamer4D [1] introduces NTA-IoU and NTL-IoU for this reason: NTA-IoU compares vehicles detected in the novel-trajectory images with boxes projected from the annotations, and NTL-IoU compares detected lanes with lanes projected from the annotations, whereas FreeVS, which our related work discusses (L446), evaluates its novel-trajectory synthesis by scoring a 3D detector on the synthesized views. Our object and lane agreement scores follow the same design as NTA-IoU and NTL-IoU, comparing detections on the generated frames with the agents and lanes projected from the annotations (Appendix L812), and they are applied online to every generated segment together with temporal consistency and distributional support, directly measuring the feature agreement, temporal consistency, and rollout survival that the reviewer notes.

The longest policy-driven rollout we tested is 20 s, and step-level reliability changes slowly along it. DriveVerse is distilled on clips extended to 200 future frames for long context (L323) and is evaluated over 20 s generation under recorded motion in the paper (Appendix L837). We also test recurrent rollouts up to 20 s with the frozen VLA policy on the retained test clips, replanning every 0.5 s and applying the online reliability check at every segment.

Horizon4 s8 s12 s16 s20 s
Full-rollout acceptance87%73%60%49%39%
Per-step acceptance98.3%98.1%97.9%97.8%97.7%

Full-rollout acceptance is cumulative because one rejected segment fails the whole rollout, so it necessarily falls with the horizon, whereas the per-step acceptance row removes that compounding and stays between 98.3% and 97.7% from 4 s to 20 s. This table also appears in our response to Reviewer K5Fi.

The 4 s training horizon is the horizon of the benchmark, and it already contains eight consecutive closed-loop decisions. Online training rollouts follow a common route for 4 s with 0.5 s replanning (L362), and because NAVSIM scores a 4 s plan where every score component and the PDM-Closed reference share that 4 s horizon (Appendix L1090), 4 s rollouts keep the training reward and the evaluation metric aligned. Training on short model-generated rollouts that branch from real data is also a standard recipe for limiting model bias [2]. The benefit on later states shows in the second stage of navhard, which evaluates the policy from follow-up states (L439): stage-2 EPDMS rises from 56.8 to 59.6 for the VLA, from 41.0 to 52.8 for the WAM, and from 46.9 to 54.2 for the E2E policy relative to their RL baselines (Table 4, L378).

[1] G. Zhao, C. Ni, X. Wang, et al. DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation. CVPR, 2025.

[2] M. Janner, J. Fu, M. Zhang, et al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS, 2019.

W3, W10, Q1, and Q20: Roles of the Simulator, the World Model, and the Reward

We thank the reviewer for these questions; this comment sets out who sees what in the interaction loop and what the paper claims.

The paper defines DriveVerse as a learned observation generator coupled to a deterministic structured simulator. Equation 1 writes the three parts: the simulator's state-transition function, the scene renderer that builds map and ego-motion conditions, and the world model that generates the camera observations (L99). The introduction says a structured simulation backend executes the leading segment of each plan and DriveVerse renders the resulting camera views (L74), and the second contribution describes a loop around a shared, deterministically updated world state (L86). The paper also states that DriveVerse requires aligned HD maps and machine-labeled scene annotations (L477). This coupling follows an established design in generative closed-loop driving simulation: DriveArena [1] pairs a traffic simulator, which executes the ego plan and moves the surrounding traffic, with a layout-conditioned generative model that renders the driving agent's camera views.

At every decision, the policy, the world model, and the reward each receive a different part of the information.

ComponentReceivesDoes not receive
Driving policyCamera views, ego state, navigation inputStructured scene state, future states, reward annotations
DriveVerseAccepted visual history; ego motion, perspective map rasters and BEV tokens rendered from the simulated state for the executed segmentThe remaining plan and all future states
Reward backendFull structured state, the proposed plan, the executed trajectoryGenerated images

The policy receives sensor history, ego state, and navigation input, while structured future states used to score candidate plans remain internal to the simulator (L133) and future states and reward annotations stay internal to the scoring backend (Appendix L868).

Scoring from the simulator state is how the benchmark defines driving quality, and the RL baselines are trained the same way. The local reward is the proposed plan's NAVSIM v2 EPDMS scored from the current structured state, keeping training aligned with evaluation (L292). The RL baselines follow this setup (L356): GRPO and FlowGRPO optimize the same PDM rewards for alternative plans from each logged observation (L434). Because this reward source is shared, it cannot explain the gains in Table 5, where the scorer, VLA initialization, scenes, sample counts, and update budgets are matched (L429). The reward differs only in the extended comfort term, which GRPO on logged observations has no second plan to compute, as in prior work; see our response to Reviewer fwbe, W1 and Q1. Replanning from generated post-action observations raises EPDMS from 91.1 to 92.1 on navtest and from 47.1 to 49.4 on navhard with the step advantage alone, compared with 91.8 and 50.0 for the trajectory advantage and 92.8 and 52.3 for T-GRPO (Table 5, L415). Our comment on W5 and W6 discusses this table further.

The world model, the diagnostic modules, and the simulation backend are all frozen throughout policy optimization. The world model is fixed during policy training (L313), all updates hold the observation generator and simulation backend fixed (Appendix L991), and checkpoints and diagnostic parameters stay fixed at evaluation (L409). Two properties protect the policy from systematic world-model bias. First, rewards never read generated pixels, so an image error cannot change a reward; the gate additionally withholds segments that disagree with the simulated state. Second, evaluation uses only official NAVSIM v2 benchmark observations, none generated by DriveVerse (L360), and all three policy families improve there, showing that what policies learn in DriveVerse carries over.

[1] X. Yang, L. Wen, T. Wei, et al. DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving. ICCV, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.