- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SimVTA post-trains a driving world action model using trajectory optimization and simulator-rendered video targets for alternative actions. Starting from a NAVSIM-adapted Epona model, it samples trajectories, executes them through PDM, renders eligible high-reward paths with World Engine, and applies flow-matching supervision to relatively high-error rendered futures. Video gradients update the shared representations used by the trajectory planner; inference requires neither rendering nor video generation. The reported results are 91.3 PDMS on NAVSIM v1, versus 86.2 for initialization and 89.1 for trajectory-only post-training; 88.2 EPDMS on NAVSIM v2; and 37.3 average HD-Score on HUGSIM.
Critical strengths
-
The core mechanism has a convincing within-backbone test. Exp0 and Exp1 both reach 89.1 PDMS, while allowing video gradients into shared representations raises the score to 91.3 (Table 5). Table 14 reports 89.1 ± 0.1 for Exp0 and 91.3 ± 0.1 for Exp2 across three seeds. This is stronger evidence for the proposed contribution than the full 5.1-point improvement over initialization, which also includes trajectory post-training.
-
Several controls directly test the value of action-matched supervision. A fixed, correctly paired offline cache achieves 90.4 versus 91.3 for online rendering, and shuffled targets reduce performance to 83.1 (Tables 8–9). The authors appropriately acknowledge that shuffling also changes supervision coverage and that some quadrant comparisons change trajectory advantages. The clean Q2 versus Q1+Q2 comparison is supported by per-seed results (Table 14). Together, these experiments make the paper informative about supervision design rather than merely introducing a combined training recipe.
Critical weaknesses
-
The reliability of the novel-action targets remains insufficiently measured. Renderer FID/FVD and model video-quality metrics are evaluated against logged ground-truth trajectories (Table 4; Appendix A.5), whereas the method's distinctive supervision comes from alternative paths executed. Safety and pose-support checks do not establish rendering accuracy under disocclusion or viewpoint shift. Selecting high prediction error could emphasize both rendering artifacts and genuinely informative futures, a limitation acknowledged in Appendix A.7. This limits the interpretation that the model is learning more accurate visual consequences of alternative actions.
-
The strongest controlled evidence is confined to one backbone and NAVSIM v1. HUGSIM compares the final model with externally reported baselines but does not evaluate the Epona initialization or Exp0 under the same closed-loop setup (Appendix A.6). Thus, it supports final-model competitiveness, not an isolated closed-loop benefit of video alignment. On NAVSIM v2, the aggregate improvement is accompanied by lower traffic-light compliance (99.7 to 99.0) and reduced comfort (67.6 to 60.0). Online rendering cost and effective supervision coverage are not quantified, so equal per-round rendering caps do not yield a favorable total-cost trade-off relative to the offline cache (Appendices A.4, A.6–A.7). These gaps limit the scope of the otherwise credible result.
Questions for authors
-
How does rendering quality vary with deviation from the logged path, and what fraction of sampled scenes/candidates actually receive video supervision? Can the high-error subset be checked for enrichment in reconstruction artifacts relative to the other rendered candidates?
-
What are the Epona-initialization and Exp0 results on the same HUGSIM routes and controller used for SimVTA? This would determine whether the measured video-supervision benefit survives closed-loop execution.
5 comments
Additional Rendering Quality Evaluation
We summarize the evaluations of 400 videos by three human raters in the table below.
| Rendering Quality | High | Moderately High | Medium | Moderately Low | Low |
|---|---|---|---|---|---|
| Rater 1 | 355 | 45 | 0 | 0 | 0 |
| Rater 2 | 346 | 54 | 0 | 0 | 0 |
| Rater 3 | 361 | 39 | 0 | 0 | 0 |
Evaluation Criteria
- High: Natural, clear visuals with intact object structures and virtually no rendering artifacts.
- Moderately High: Good visual quality with minor edge blurring that does not affect overall appearance.
- Medium: Noticeable local blurring, while major objects and the scene remain clearly recognizable.
- Moderately Low: Blurring and missing textures in multiple regions, reducing visual quality and making objects harder to recognize.
- Low: Severe distortions, missing content, or deformations, making objects and the scene difficult to recognize.
Thank you again for your valuable comments. We hope our responses have effectively addressed the concerns raised. If there are any remaining concerns, we would be grateful for further clarification
Response to Reviewer Reyt
Thank you for your positive assessment of SimVTA. We have added the following analyses and experiments.
1. Rendering reliability
Novel-view rendering quality depends on the spatial coverage of the 3D assets. The asset region is defined by the logged trajectory and curves offset by 20 m to either side. Our 3DGS assets are reconstructed using MTGS [1], whose Figure 1 shows an example that maintains high visual quality at a lateral offset of 8 m, approximately two lanes. Trajectories sampled during reinforcement learning generally deviate only slightly from the logged trajectory, and we also cap their maximum deviation at 8 m to minimize rendering errors. To further assess rendering reliability, we randomly selected 100 training scenes and rendered videos for 400 sampled trajectories using the training configuration. The maximum trajectory deviation was 5.1 m. The deviation distribution is shown below.
Ground-truth future videos are unavailable for alternative trajectories, so their rendering reconstruction errors cannot be measured directly. We asked three raters to independently inspect all 400 videos and observed no decline in rendering quality as deviation increased within the range used in training. The broad spatial coverage of the 3D assets and the manual inspection results together support rendering reliability.
SimVTA's trajectory reward is based on NAVSIM v1 scores and does not include NAVSIM v2's traffic-light compliance or two_frame_extended_comfort scores. Our model has also not been fine-tuned on NAVSIM v2, which accounts for the decrease in these two scores. SimVTA also achieves state-of-the-art performance in closed-loop evaluation on HUGSIM, which better demonstrates the importance of learning from video during post-training.
Online rendering takes 23 hours, versus 25 hours for a fixed offline cache (8 for cache construction and 17 for post-training), and achieves higher PDMS (91.3 vs. 90.4).
Thank you for your suggestions. We hope these clarifications and additional experiments address your main concerns. We are happy to clarify any remaining questions.
[1] 2025, arXiv, MTGS: Multi-Traversal Gaussian Splatting