acceptodds
Under review as a conference paper at ICLR 2027

SimVTA: Simulation-Grounded Video–Trajectory Alignment for Driving World Action Models

Abstract

Driving world action models (WAMs) learn from paired trajectories and future observations. Post-training with reinforcement learning allows the policy to explore alternative trajectories and improve planning using driving rewards. However, driving logs contain future videos only for the recorded trajectories, leaving these alternatives without corresponding visual supervision. Thus, we introduce SimVTA, a simulation-grounded post-training framework that combines trajectory optimization with video supervision. First, we sample candidate trajectories for each observation and use a real-world simulator (e.g., World Engine) to render future videos for high-reward candidates in real-world scenes reconstructed with 3D Gaussian Splatting (3DGS). This provides trajectory-matched supervision independent of the model's own predictions. Under a limited rendering budget, we prioritize video supervision for higher-error futures among these rendered candidates. Finally, we optimize trajectory planning with Group Relative Policy Optimization (GRPO) and use a flow-matching loss on the matched videos to update shared representations. At inference time, the model predicts trajectories directly, without simulator calls. On NAVSIM v1, applying SimVTA to a WAM initialization improves PDMS from 86.2 to 91.3, a gain of 5.1 points. It achieves 88.2 EPDMS on NAVSIM v2 and an average HD-Score of 37.3 in closed-loop evaluation on HUGSIM. Ablations show a clear planning benefit from allowing video supervision to update shared representations and suggest an additional benefit from prioritizing higher-error futures among high-reward trajectories.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous driving, World action models, Reinforcement learning, Video-trajectory alignment, Simulation-based learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ReytOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SimVTA post-trains a driving world action model using trajectory optimization and simulator-rendered video targets for alternative actions. Starting from a NAVSIM-adapted Epona model, it samples trajectories, executes them through PDM, renders eligible high-reward paths with World Engine, and applies flow-matching supervision to relatively high-error rendered futures. Video gradients update the shared representations used by the trajectory planner; inference requires neither rendering nor video generation. The reported results are 91.3 PDMS on NAVSIM v1, versus 86.2 for initialization and 89.1 for trajectory-only post-training; 88.2 EPDMS on NAVSIM v2; and 37.3 average HD-Score on HUGSIM.

Critical strengths
  1. The core mechanism has a convincing within-backbone test. Exp0 and Exp1 both reach 89.1 PDMS, while allowing video gradients into shared representations raises the score to 91.3 (Table 5). Table 14 reports 89.1 ± 0.1 for Exp0 and 91.3 ± 0.1 for Exp2 across three seeds. This is stronger evidence for the proposed contribution than the full 5.1-point improvement over initialization, which also includes trajectory post-training.

  2. Several controls directly test the value of action-matched supervision. A fixed, correctly paired offline cache achieves 90.4 versus 91.3 for online rendering, and shuffled targets reduce performance to 83.1 (Tables 8–9). The authors appropriately acknowledge that shuffling also changes supervision coverage and that some quadrant comparisons change trajectory advantages. The clean Q2 versus Q1+Q2 comparison is supported by per-seed results (Table 14). Together, these experiments make the paper informative about supervision design rather than merely introducing a combined training recipe.

Critical weaknesses
  1. The reliability of the novel-action targets remains insufficiently measured. Renderer FID/FVD and model video-quality metrics are evaluated against logged ground-truth trajectories (Table 4; Appendix A.5), whereas the method's distinctive supervision comes from alternative paths executed. Safety and pose-support checks do not establish rendering accuracy under disocclusion or viewpoint shift. Selecting high prediction error could emphasize both rendering artifacts and genuinely informative futures, a limitation acknowledged in Appendix A.7. This limits the interpretation that the model is learning more accurate visual consequences of alternative actions.

  2. The strongest controlled evidence is confined to one backbone and NAVSIM v1. HUGSIM compares the final model with externally reported baselines but does not evaluate the Epona initialization or Exp0 under the same closed-loop setup (Appendix A.6). Thus, it supports final-model competitiveness, not an isolated closed-loop benefit of video alignment. On NAVSIM v2, the aggregate improvement is accompanied by lower traffic-light compliance (99.7 to 99.0) and reduced comfort (67.6 to 60.0). Online rendering cost and effective supervision coverage are not quantified, so equal per-round rendering caps do not yield a favorable total-cost trade-off relative to the offline cache (Appendices A.4, A.6–A.7). These gaps limit the scope of the otherwise credible result.

Questions for authors
  1. How does rendering quality vary with deviation from the logged path, and what fraction of sampled scenes/candidates actually receive video supervision? Can the high-error subset be checked for enrichment in reconstruction artifacts relative to the other rendered candidates?

  2. What are the Epona-initialization and Exp0 results on the same HUGSIM routes and controller used for SimVTA? This would determine whether the measured video-supervision benefit survives closed-loop execution.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Reyt

Thank you for your positive assessment of SimVTA. We have added the following analyses and experiments.

Question 1

1. Rendering reliability

Novel-view rendering quality depends on the spatial coverage of the 3D assets. The asset region is defined by the logged trajectory and curves offset by 20 m to either side. Our 3DGS assets are reconstructed using MTGS [1], whose Figure 1 shows an example that maintains high visual quality at a lateral offset of 8 m, approximately two lanes. Trajectories sampled during reinforcement learning generally deviate only slightly from the logged trajectory, and we also cap their maximum deviation at 8 m to minimize rendering errors. To further assess rendering reliability, we randomly selected 100 training scenes and rendered videos for 400 sampled trajectories using the training configuration. The maximum trajectory deviation was 5.1 m. The deviation distribution is shown below.

Deviation (m)DistributionCount
0.0–0.5█3
0.5–1.0██████████████68
1.0–1.5████████████████80
1.5–2.0█████████████████████106
2.0–2.5█████████47
2.5–3.0███████████54
3.0–3.5████21
3.5–4.0███14
4.0–4.5█2
4.5–5.0█3
5.0–5.5█2

Ground-truth future videos are unavailable for alternative trajectories, so their rendering reconstruction errors cannot be measured directly. We asked three raters to independently inspect all 400 videos and observed no decline in rendering quality as deviation increased within the range used in training. The broad spatial coverage of the 3D assets and the manual inspection results together support rendering reliability.

  1. Supervision coverage For each scene, we sample 32 trajectories, render up to four eligible high-reward candidates, and apply video supervision to those with prediction errors at or above the within-scene mean. Training statistics show that 94.7% of sampled scenes and 5.9% of all sampled trajectories receive video supervision.
  2. High-error subset For the 100 randomly selected scenes, we identified the high-error subset using the training configuration. The manual inspection described above found no obvious rendering artifacts in either this subset or the remaining candidates. Comparing the rendered videos with the baseline model's predictions indicated that the high errors mainly arose from poor predictions. These are precisely the futures the model needs to learn to predict.

Question 2

  1. Closed-loop improvement We additionally evaluated the Epona initialization and Exp0 on HUGSIM under the same configuration as SimVTA. They achieve HD-Scores of 28.2 and 31.4, respectively, with detailed results below. These scores are 9.1 and 5.9 points below SimVTA's score of 37.3, providing evidence that the planning gains from video supervision carry over to closed-loop execution. We have added these results to Table 16 in the appendix.
MethodRC ERC MRC HRC XRC Avg.HD-Score EHD-Score MHD-Score HHD-Score XHD-Score Avg.
Epona58.049.628.623.239.147.340.115.612.128.2
Exp070.748.930.135.044.061.435.816.023.131.4
SimVTA78.856.138.539.650.969.742.322.925.737.3

Weakness

  1. NAVSIM v2

SimVTA's trajectory reward is based on NAVSIM v1 scores and does not include NAVSIM v2's traffic-light compliance or two_frame_extended_comfort scores. Our model has also not been fine-tuned on NAVSIM v2, which accounts for the decrease in these two scores. SimVTA also achieves state-of-the-art performance in closed-loop evaluation on HUGSIM, which better demonstrates the importance of learning from video during post-training.

  1. Compute cost

Online rendering takes 23 hours, versus 25 hours for a fixed offline cache (8 for cache construction and 17 for post-training), and achieves higher PDMS (91.3 vs. 90.4).

MethodTotal time (h)PDMS
Trajectory-only post-training1889.1
Fixed offline cache25 (8 + 17)90.4
Online rendering (SimVTA)2391.3

Thank you for your suggestions. We hope these clarifications and additional experiments address your main concerns. We are happy to clarify any remaining questions.

[1] 2025, arXiv, MTGS: Multi-Traversal Gaussian Splatting

AuthorsOpenReview ↗

Additional Rendering Quality Evaluation

We summarize the evaluations of 400 videos by three human raters in the table below.

Rendering QualityHighModerately HighMediumModerately LowLow
Rater 135545000
Rater 234654000
Rater 336139000

Evaluation Criteria

  • High: Natural, clear visuals with intact object structures and virtually no rendering artifacts.
  • Moderately High: Good visual quality with minor edge blurring that does not affect overall appearance.
  • Medium: Noticeable local blurring, while major objects and the scene remain clearly recognizable.
  • Moderately Low: Blurring and missing textures in multiple regions, reducing visual quality and making objects harder to recognize.
  • Low: Severe distortions, missing content, or deformations, making objects and the scene difficult to recognize.

Thank you again for your valuable comments. We hope our responses have effectively addressed the concerns raised. If there are any remaining concerns, we would be grateful for further clarification

Reviewer_yDSSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SimVTA addresses how to supervise future-image prediction when a world-action model is optimized with GRPO. It renders images along sampled trajectories using Gaussian splatting, providing visual targets beyond the logged expert trajectory.

Critical strengths

The idea is intuitive and addresses a meaningful problem: policy exploration produces trajectories for which recorded future images are unavailable. Rendering trajectory-conditioned supervision is a practical way to maintain joint visual and action learning during reinforcement learning.

Critical weaknesses
  1. Gaussian-rendered images can have a large gap from real images when trajectories move outside the recorded views. Directly surpervision using this may cause problem. This problem does not seem sufficiently addressed in the paper.
  2. Gaussian scene reconstruction and rendering can be time-consuming. Every trajectory need generate a Guassian scene, it can be very resource consuming, if the model performance don't have a a significant improvement,it may be not very valuble. The paper lacks a clear analysis of this overhead, raising concerns about practical utility.
  3. The comparisons seem selective and miss some strong baselines. For example, DrivoR achieves 93.1 on NAVSIM v1, higher than SimVTA’s 91.3. The paper also omits related work such as WA-JEPA, which reports 91.7 EPDMS, although the evaluation versions need to be checked for a fair comparison.
Questions for authors

Can you evaluate rendering quality as sampled trajectories move further from the recorded route, and show how rendering artifacts affect training? What are the scene-reconstruction cost, online-rendering overhead, and total training cost relative to trajectory-only GRPO? Can you compare with DrivoR and WA-JEPA under matched data and evaluation protocols, particularly the corrected NAVSIM v2 evaluator?

5 comments
AuthorsOpenReview ↗

Response to Reviewer yDSS

Thank you for your positive assessment of SimVTA. We have added the following analyses and experiments.

Question 1. Rendering reliability

Novel-view rendering quality depends on the spatial coverage of the 3D assets. The asset region is defined by the logged trajectory and curves offset by 20 m to either side. Our 3DGS assets are reconstructed using MTGS [1], whose Figure 1 shows an example that maintains high visual quality at a lateral offset of 8 m, approximately two lanes. Trajectories sampled during reinforcement learning generally deviate only slightly from the logged trajectory, and we also cap their maximum deviation at 8 m to minimize rendering errors.

To further assess rendering reliability, we randomly selected 100 training scenes and rendered videos for 400 sampled trajectories using the training configuration. The maximum trajectory deviation was 5.1 m. The deviation distribution is shown below.

Deviation (m)DistributionCount
0.0–0.5█3
0.5–1.0██████████████68
1.0–1.5████████████████80
1.5–2.0█████████████████████106
2.0–2.5█████████47
2.5–3.0███████████54
3.0–3.5████21
3.5–4.0███14
4.0–4.5█2
4.5–5.0█3
5.0–5.5█2

Ground-truth future videos are unavailable for alternative trajectories, so their rendering reconstruction errors cannot be measured directly. We asked three raters to independently inspect all 400 videos; none showed rendering blur or artifacts. The broad spatial coverage of the 3D assets and the manual inspection results together support rendering reliability.

Question 2. Rendering cost

3DGS rendering does incur substantial overhead. SimVTA takes 23 hours to train, compared with 18 hours for trajectory-only reinforcement learning, as reported in the table below. The additional time is mainly spent loading the 3D assets for each scene and rendering videos. We reuse the publicly released 3DGS scene assets from World Engine [2], so SimVTA requires no additional scene reconstruction.

MethodTotal time (h)PDMS
Trajectory-only post-training1889.1
Fixed offline cache25 (8 + 17)90.4
Online rendering (SimVTA)2391.3

Question 3. Comparison with DrivoR and WA-JEPA

We evaluated DrivoR and WA-JEPA on NAVSIM v2 using the same data and evaluation protocol; they achieve scores of 88.7 and 91.5, respectively. These are strong models, while SimVTA's main contribution is a post-training framework that uses video supervision to improve trajectory planning. Our focus is therefore on the improvement over the baseline after applying this framework. SimVTA has also not been fine-tuned on the NAVSIM v2 reward.

SimVTA also achieves state-of-the-art performance in closed-loop evaluation on HUGSIM. Compared with open-loop evaluation on NAVSIM v2, closed-loop evaluation better assesses the model’s ability to recover from errors during long-horizon driving. These results highlight the importance of learning from video during post-training.

We additionally evaluated the Epona initialization and Exp0 on HUGSIM under the same configuration as SimVTA. They achieve HD-Scores of 28.2 and 31.4, respectively, with detailed results below. These scores are 9.1 and 5.9 points below SimVTA's score of 37.3, providing evidence that the planning gains from video supervision carry over to closed-loop execution. We have added these results to Table 16 in the appendix.

MethodRC ERC MRC HRC XRC Avg.HD-Score EHD-Score MHD-Score HHD-Score XHD-Score Avg.
Epona58.049.628.623.239.147.340.115.612.128.2
Exp070.748.930.135.044.061.435.816.023.131.4
SimVTA78.856.138.539.650.969.742.322.925.737.3

Thank you for your suggestions. We hope these clarifications and additional experiments address your main concerns. We are happy to clarify any remaining questions.

[1] 2025, arXiv, MTGS: Multi-Traversal Gaussian Splatting

[2] 2026, arXiv, World Engine: Towards the Era of Post-Training for Autonomous Driving

AuthorsOpenReview ↗

Additional Rendering Quality Evaluation

We summarize the evaluations of 400 videos by three human raters in the table below.

Rendering QualityHighModerately HighMediumModerately LowLow
Rater 135545000
Rater 234654000
Rater 336139000

Evaluation Criteria

  • High: Natural, clear visuals with intact object structures and virtually no rendering artifacts.
  • Moderately High: Good visual quality with minor edge blurring that does not affect overall appearance.
  • Medium: Noticeable local blurring, while major objects and the scene remain clearly recognizable.
  • Moderately Low: Blurring and missing textures in multiple regions, reducing visual quality and making objects harder to recognize.
  • Low: Severe distortions, missing content, or deformations, making objects and the scene difficult to recognize.

Thank you again for your valuable comments. We hope our responses have effectively addressed the concerns raised. If there are any remaining concerns, we would be grateful for further clarification

Reviewer_gQC4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies post-training for WAMs. Observing that existing work mostly overlooks the potential of exploring visual supervision of future videos. They proposed a simulation-grounded post-training framework that combines trajectory optimization with video supervision. Overall, this paper addresses a meaningful limitation of logged driving data, the proposed framework is coherent, and the same-backbone ablations provide encouraging evidence that simulator-matched video supervision improves planning. However, I have concerns about the novelty, optimization formulation, and interpretation of the numerical results.

The evaluation of the proposed methods on NAVSIM v1, NAVSIM v2, and closed-loop HUGSIM datasets are reasonable and sufficiently comprehensive. The numerical improvements seem reasonable and significant.

Critical strengths
  • the problem is timely and the paper is overall well-written 

  • The motivation for obtaining matched visual targets for alternative trajectories is clear

  • The shared-representation ablation is informative: adding video supervision raises NAVSIM v1 PDMS from 89.1 to 91.3, whereas updating only the video head provides no additional planning benefit 

  • Evaluation includes NAVSIM v1, NAVSIM v2, and closed-loop HUGSIM. Selected NAVSIM v1 ablations are repeated across three random seeds.

Critical weaknesses
  • novelty needs to be better justified. Simulator-based post-training, action-conditioned video supervision, shared representations, and flow matching are established and relatively standard components. The contribution appears to be their integration. The paper should distinguish this contribution more explicitly from World Engine, ReSim, and existing video–planning co-training approaches. The high-reward/high-error selection rule is relatively simple, and its measured advantage over supervising all high-reward rendered candidates is only approximately 0.3 PDMS.

  • inconsistent optimization objective: The mainbody of the paper describes trajectory optimization as GRPO, but Appendix A.3 explicitly states that the implemented objective uses no current-to-sampling-policy probability ratio. Instead, it weights clipped and dimension-averaged Gaussian transition scores by group-relative advantages. 

- The increase from 86.2 to 91.3 PDMS includes trajectory post-training, which alone reaches 89.1. The more direct estimate of the additional benefit of video supervision is therefore 2.2 points. This remains a useful improvement, but should be distinguished from the total 5.1-point gain. Several selection ablations also change supervision counts, optimization objectives, or safety treatment, limiting attribution to the selection rule alone.

Questions for authors

none

4 comments
AuthorsOpenReview ↗

Response to Reviewer gQC4

Thank you for your careful review and suggestions. We have added the following analyses and experiments.

Weakness 1

  1. Clarification of novelty

    Problem-level novelty. We identify a trajectory–video supervision mismatch in driving WAM post-training: policy optimization explores alternative trajectories, but logged future videos correspond only to recorded trajectories.

    Method-level novelty. We introduce simulation-grounded video–trajectory alignment, rendering matched visual targets for explored trajectories and using video supervision to improve planning through shared representations.

  2. Discussion of related work ReSim and World Engine focus on building and using simulators, whereas SimVTA focuses on driving policy post-training. ReSim uses simulated videos to score candidates at inference time; World Engine provides trajectory rewards through simulation, but the model does not learn from visual supervision corresponding to those trajectories. Following your suggestion, we have clarified SimVTA's contribution and its differences from World Engine, ReSim, and existing video-planning co-training methods. We have uploaded the revised PDF with changes highlighted in blue; please see Appendix A.3, “Novelty and Relation to Prior Work.”

  3. Limited improvement Compared with supervising all rendered candidates, SimVTA uses approximately half as many video-supervised candidates, reduces training time by 1 hour, and improves PDMS by 0.3 points. This selection strategy improves both planning performance and computational efficiency. This finding motivates further research on high-error video supervision.

MethodVideo-supervised candidates per sceneTotal time (h)PDMS
Supervising all rendered candidates≈42491.0
SimVTA≈22391.3

Weakness 2

  1. Inconsistency in the optimization objective Thank you for pointing this out. We use SDE sampling from Flow-GRPO [1] and an objective based on DiffGRPO in ReCogDrive [2]: group-relative advantages weight clipped, dimension-averaged transition scores, without a policy probability ratio. We have renamed this procedure “group-relative trajectory optimization” and revised both the main text and appendix to consistently describe the implemented objective and its differences from standard GRPO. These revisions correct only the terminology and description; the experimental results remain unchanged.

Weakness 3

  1. Performance gains Trajectory-only post-training improves PDMS from 86.2 to 89.1. Our trajectory-matched video supervision further improves PDMS to 91.3, yielding an additional 2.2-point gain and a total improvement of 5.1 points over initialization. We have revised the relevant passages to clarify these incremental gains.
  2. Selection ablations Thank you for identifying this issue in the selection ablations. We have added Exp10, which uses the same amount of supervision as SimVTA and replaces only high-error selection with random video selection from the same high-reward candidate pool. Across three random seeds, this control achieves 90.9 PDMS, compared with SimVTA's 91.3, supporting an additional benefit of high-error selection over random selection. The results are reported in Tables 6 and 13 and reproduced below.
MethodNCDACTTCComfortEPPDMS
Exp1097.797.096.799.986.390.9
SimVTA98.097.297.0100.086.791.3

Thank you for your suggestions. We hope these clarifications and additional experiments address your main concerns. If these clarifications address your concerns, we would be grateful if you would consider raising your rating. We are happy to clarify any remaining questions.

[1] 2025, NeurIPS, Flow-GRPO: Training Flow Matching Models via Online RL

[2] 2026, ICLR, ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

Thank you again for your valuable comments. We hope our responses have effectively addressed the concerns raised. As the rebuttal period is ending soon, we’d sincerely appreciate a score update if the concerns are resolved. If there are any remaining concerns that prevent a higher score, we would be grateful for further clarification.

Thank you very much for the clarification on the novelty, and on the inconsistency of the objective. The additional experiments are insightful, making the potential of the proposed methods convincing. Hence, most of my concerns have been resolved. I updated my rating.

Thank you very much for your thoughtful feedback and for updating your rating. We sincerely appreciate your time and constructive suggestions, which have helped us improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.