acceptodds
Under review as a conference paper at ICLR 2027

Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving via Action-Faithful Rollouts

Abstract

End-to-end autonomous driving policies are commonly trained via imitation learning on logged demonstrations without observing the consequences of their own actions, leading to causal confusion in closed-loop real-world deployment. To address this issue, reinforcement learning (RL) post-training offers a promising alternative by leveraging world models as interactive training environments to enable future scene generation for policy improvement. Nevertheless, existing approaches either rely on reconstruction-based simulators, offering limited counterfactual interaction, or adopt synthetic simulators to enable long-horizon closed-loop interaction at the cost of a substantial sim-to-real gap. Recently, video world models have exhibited the ability to generate realistic multi-step future rollouts but may not faithfully reflect action conditions, resulting in action-vision mismatch. In this paper, we introduce RoXDrive, a plug-and-play closed-loop RL framework that enables reliable policy optimization by identifying action-faithful world-model rollouts, consisting of two stages: 1) Model pre-training: In addition to imitation-based policy pre-training, we devise an Action-Vision Faithfulness Evaluator for inverse dynamics estimation with our geometry-aware auxiliary trajectory supervision, enabling long-horizon assessment of whether visual dynamics faithfully reflect the conditioning ego actions. 2) Action-faithful RL post-training: Agents iteratively interact with world models to form long-horizon scene rollouts, retaining only action-faithful ones for dense safety-aware scoring and scene-level closed-loop RL post-training. Extensive experiments on nuScenes and an in-house dataset with over 130K training scenarios demonstrate consistent gains across planners, reducing safety violations by 27.6% with DiffusionDrive on nuScenes and 33.7% with Qwen3-VL on the internal data. Code will be released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u9SkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RoXDrive performs closed-loop RL post-training of driving policies using an action-conditioned video world model. An inverse-dynamics evaluator with geometry-aware supervision filters rollouts whose visual ego motion disagrees with the conditioning actions. Experiments on nuScenes and an internal dataset report improved safety and driving scores across multiple planners within X-World.

Critical strengths

The paper tries to address a consequential problem: visually plausible world-model rollouts may violate action conditions and provide misleading policy-training signals. The experiments and filtering ablations provide useful evidence that geometry-aware supervision improves inverse dynamics and yields additional gains within the tested environment.

Critical weaknesses
  1. Closed-loop training and evaluation both depend on the closed-source X-World, without a demonstrated reproduction path or independent-environment validation. The results establish simulator-specific improvements, while their transferability remains uncertain.
  2. Inverse-dynamics-based rollout checking has close precedents, including SC3-Eval. The remaining contribution requires stronger comparisons against simple consistency filtering and clearer separation from additional expert-supervised hard-scene training. Experiment scale alone does not establish this distinction.
Questions for authors
  1. What access, checkpoints, and evaluation resources will be provided to enable independent reproduction of the core closed-loop results?
  2. How is AVFE validated on generated videos, particularly regarding false acceptance, false rejection, and retention of difficult or unsafe scenarios?
  3. Do the gains persist against matched-retention consistency-filtering baselines and equivalent additional expert supervision? Independent-environment results would also strengthen my assessment.
1 comment

Thank you for the additional experiments and clarifications. They partially address concerns. However, several concerns remain:

  1. The HUGSIM results compare the original policies with full RoXDrive. Evaluating the Table 5 ablation checkpoints in HUGSIM would help establish whether AVFE’s incremental benefit also transfers.
  2. Reproducibility: The proposed HUGSIM-based release is encouraging, but it has not yet been shown to reproduce the central findings obtained using the closed-source X-World.
  3. Overall retention rates do not establish filtering accuracy or exclude selective removal of difficult actions. The paired analysis is useful, but does not replace comparisons controlling retention and expert-supervision budgets over training.

The rebuttal strengthens the empirical evidence but only partially resolves my concerns. I therefore maintain my original rating.

Reviewer_4VUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoXDrive, a closed-loop RL framework using action-conditioned video world models. It introduces an Action-Vision Faithfulness Evaluator (AVFE) to filter unreliable rollouts, followed by safety-aware scoring and scene-level GRPO. Experiments demonstrate improvements across multiple driving architectures on nuScenes and an in-house dataset.

Critical strengths

1.The paper addresses an important issue in world-model-based RL. Geometry-aware supervision effectively reduces accumulated errors in inverse dynamics estimation. 2.The method demonstrates consistent improvements across multiple planners and VLAs, with ablations supporting the effectiveness of AVFE-based filtering.

Critical weaknesses
  1. Action-vision faithfulness does not guarantee reliable simulation. AVFE only verifies ego-motion consistency, without evaluating the physical plausibility of surrounding-agent behaviors or counterfactual interactions. Consequently, filtered rollouts may still provide misleading safety rewards.
  2. The closed-loop evaluation lacks independent validation. Both training and evaluation rely on X-World. Moreover, public nuScenes evaluation covers only 4 seconds and two action steps, while longer-horizon experiments use a private dataset. This limits the evidence for generalization beyond the training simulator.
  3. The potential selection bias introduced by rollout filtering is insufficiently analyzed. The paper does not report rollout retention rates or the characteristics of rejected episodes. Filtering may disproportionately discard challenging scenarios, reducing the diversity of safety-critical experiences available for RL.
  4. The computational complexity and practical applicability are insufficiently discussed. The framework relies on expensive video generation and a closed-source world model. Some planner adaptations also require additional trajectory-selection modules, which weakens the claim of being fully plug-and-play.
Questions for authors
  1. How is the physical consistency of surrounding agents and counterfactual interactions verified beyond ego-motion faithfulness?
  2. Can the trained policies be evaluated using an independent simulator or world model?
  3. What are the rollout retention rates, and how does filtering affect the distribution of challenging or safety-critical scenarios?
  4. What is the computational overhead of the proposed framework compared with conventional RL post-training?
Reviewer_RduqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoXDrive, a closed-loop RL framework for end-to-end autonomous driving using a frozen video world model. Its action-vision faithfulness evaluator uses geometry-aware trajectory supervision to assess action consistency and filter unreliable rollouts. Experiments on nuScenes and large-scale in-house data show promising improvements. However, RL adaptation across planners, training rollout length, independent evaluation, and simulator efficiency and interaction realism remain insufficiently addressed.

Critical strengths
  1. The paper explores an important and timely direction: using video world models for closed-loop RL in autonomous driving. The framework requires no world-model inference at deployment and shows promising results on nuScenes and large-scale in-house data.

  2. The action-vision faithfulness evaluator addresses an important source of simulator error. Geometry-aware trajectory supervision improves long-horizon inverse dynamics estimation, and ablations demonstrate the benefit of filtering unreliable rollouts.

Critical weaknesses
  1. The method is evaluated on open-loop nuScenes and an in-house dataset. Although closed-loop evaluation is constructed using X-World, it relies on the same adapted, closed-source simulator employed during RL training, raising concerns about overfitting to simulator-specific dynamics or exploiting systematic modeling errors. Validation on commonly used public benchmarks, such as the pseudo-closed-loop NAVSIM [1] or independent closed-loop simulators such as HUGSIM [2], would provide more credible evidence of policy improvement. The current evaluation limits reproducibility and fair comparison, as the results are difficult for others to reproduce, verify, or follow up on.

  2. The paper lacks key implementation details on how different policies are adapted to the proposed RL algorithm.

    • For DiffusionDrive, which combines anchor selection with diffusion-based trajectory refinement, it is unclear how action log-probabilities are computed and whether RL optimizes only anchor-selection probabilities or also the diffusion process.
    • For TransFuser, which is originally a deterministic regression-based planner, it is unclear how GRPO is applied. Any modification to the original model, such as replacing the regression planning head with a probabilistic planning head, should be explicitly described.
    • For GTRS, which contains a reward head, it is unclear which parts of the model are optimized during RL, e.g., whether RL is applied only to the imitation head or also to the reward head.
  3. The paper does not justify the number of closed-loop rollout steps used during GRPO training, although this design choice is central to both training stability and computational cost.

  4. The paper provides insufficient analysis of using the video world model as an environment.

    • It is unclear how efficient world-model inference is and whether it can support large-scale parallel rollouts.
    • It is unclear how other agents are controlled or propagated under the BEV-layout-conditioned world model, which is crucial for assessing whether the environment provides genuine interactive closed-loop feedback.
  5. The advantages of using a frozen video world model over other simulation engines remain insufficiently justified. Although AVFE addresses ego-action consistency, it does not establish temporal/multi-view consistency, scene realism, or realistic agent interactions.

    • Lines 78–83 criticize reconstruction-based simulators for limited reactive agent behavior. However, given the BEV-layout conditioning, the paper should clarify whether agent reactions are generated by the world model or externally specified through updated layouts.
    • Lines 171–172 describe 3DGS-based reconstruction as computationally prohibitive, but no inference latency, rollout throughput, or overall training cost is reported for the proposed framework to support this comparison.
Questions for authors
  1. Can the authors provide evaluation on NAVSIM or an independent simulator such as HUGSIM to verify generalization beyond X-World?

  2. How are action log-probabilities computed for DiffusionDrive and TransFuser, and which modules are updated during RL, including GTRS’s reward head?

  3. How is the number of closed-loop rollout steps chosen during GRPO training?

  4. What are the world-model inference latency, parallel rollout throughput, and total training cost?

  5. Are surrounding-agent reactions generated by the world model or externally specified through BEV layouts?

  6. Beyond ego-action consistency, what evidence supports simulator reliability and the claimed advantages over reconstruction-based simulation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.