acceptodds
Withdrawn from ICLR 2027

VITAL: From Visual Motion to Precise Action

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World-action models aim to turn visual anticipation into driving decisions, yet typically train action experts on trajectory representations defined independently of visual dynamics. This motivates deriving action targets directly from visual motion. We first examine a simpler setting in which trajectories are reconstructed from recorded future images. Even without the need for future prediction, reconstruction captures the dominant maneuver but leaves centimeter-scale errors. We therefore introduce VITAL (Vision-Induced Tokens for Action Learning), an action tokenizer that couples a visually derived motion scaffold with a compact residual encoding the trajectory error left by that scaffold. This representation makes visual motion the principal prediction target and gives small metric corrections a separate, normalized learning scale. Using flow matching, the action expert jointly predicts both components from the current image, ego state, and navigation command, without generating future images at inference. The same residual coordinates support bounded reinforcement post-training that refines execution while keeping the predicted visual scaffold fixed. With only one action denoising step, VITAL achieves 90.8 PDMS on NAVSIM v1 navtest, within 0.13 points of its ten-step score at the end-to-end inference speed. Without retraining, it achieves 91.1 EPDMS on NAVSIM v2 navtest and 38.6 on navhard.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, World Action Models, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TjAQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents an effective action representation and local residual refinement. However, the paper does not sufficiently explain why the decomposition of the visual motion scaffold and the residual improves performance. Moreover, the limited testing on the nuplan (navsim) dataset cannot fully support the claimed benefits of the action generation.

Critical strengths
  1. The paper focuses a key problem in current WAM models: the gap between the action and video modalities. It proposes a residual-based design to improve action generation.

  2. The experiments discuss most issues in detail and present clear results. The experimental analysis is fairly complete.

  3. The paper has clear figures. The animations on the anonymous website make the main design easy to understand. The 3D architecture diagram in Figure 3 is memorable.

Critical weaknesses
  1. The title is too broad. First, the experiments only cover autonomous driving. The paper does not discuss action generation in other domains, such as robot navigation and manipulation, so the title should narrow its scope. Second, the paper addresses the gap between action and video only by improving action generation. It does not improve video generation or the WAM architecture itself. This makes “visual motion” somewhat unclear, and a more specific title would help.

  2. The main idea is to add a residual module that corrects a trajectory scaffold derived from visual information while preserving the original scaffold. The key question is why this extra module is needed. Its inputs are largely the same as those of the action expert, and it does not introduce additional prior information. The paper should therefore compare this design with directly improving the original WAM action expert or trajectory tokenization. It should explain the limits of these alternatives and provide corresponding baselines. The need for the extra residual module has not yet been clearly demonstrated. Moreover, the contribution of the residual design is not new. We have seen a very similar design in ResAD and PLD (Probe, Learn, Distill), and the VITAL paper seems like an implementation under WAM.

  3. The paper focuses on action refinement. This should be especially useful on navhard, where precise trajectories matter and small deviations can cause collisions. However, RL does not improve performance on navhard. Increasing the number of training steps after RL even reduces performance.

  4. The paper does not change the WAM architecture itself. Its main improvements are in action generation and refinement, so testing generalization across different datasets is especially important. The evaluation should include datasets such as WOD-E2E or closed-loop benchmarks such as Bench2Drive, rather than only nuPlan.

  5. The paper also notes that the residual module cannot fully correct a scaffold when its error is too large. Does this suggest that correcting only the action modality is insufficient for WAM? Improving both video generation and action generation during training may be necessary to achieve better action outputs, instead of focusing only on action refinement.

Questions for authors

Please refer to the weaknesses section. I am willing to raise the score if the authors can address my concerns.

4 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer TjAQ,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer TjAQ,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer TjAQ,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 15

Thank you for your response and the additional experimental results. However, my main concern remains unresolved.

My first two questions were about the motivation for this paper and why the authors chose this method. SimWAM itself can achieve similar performance with multiple steps, so the proposed method is mainly about speeding up inference. This complex approach to training and speeding up inference is not itself new, which makes it especially important to show how well it performs, how well it generalizes, and why it is needed.

  1. If the paper’s main contribution is in action generation, then it needs to show that applying the proposed action generation method to different backbones improves performance on each of them. This kind of generalization needs to be demonstrated.

  2. Since the paper focuses on improving action generation, it needs to show that the proposed method outperforms the original backbone across different datasets. With the current results, it is hard to tell whether the strong performance simply comes from the backbone itself.

  3. The newly added experiments still focus on open-loop evaluation. If the paper emphasizes improvements in action generation, then validation in a closed-loop setting is even more important.

Reviewer_kaCAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This driving-focused paper observes that world-action models, despite shared attention, decouple how the model predicts video and action; however, predicting action trajectories directly from (perfect) video is also not perfect. Therefore, they propose to both preserve the motion / trajectory information from vision while formulating a representation for residual corrections to this trajectory information. Their method, VITAL, proposes formulating this representation via a PCA of the residual error in the predicted trajectory decoded from the visual motion features. The downstream world-action model then predicts the visual motion features used for trajectory decoding, these residual tokens, and the normal visual tokens. Finally, a policy refinement stage is used to refine the residual token predictions. Their results on NAVSIM, a simulated driving benchmark, shows that VITAL can achive good performance with only 1 denoising step.

Critical strengths
  1. The reconstruction analysis in the introduction is well-motivating for the residual representation.
  2. The one-step performance on NAVSIM v1 performs well relative to baselines.
Critical weaknesses
  1. Many terms, such as vision scaffold, visual code, motion, motion codes, residuals, etc., are not well-defined and are sometimes use interchangeably, making it very difficult to understand.
  2. VGGT-Omega is used in Figure 3, but never mentioned in the main paper. Other implementation details around the architecture and training are also not mentioned.
  3. The method introduces lots of moving parts, but the results are not that much clearly better. For instance, SimWAM is better on NAVSIM v1. If the benefits are on NFE / efficiency, then that is not currently well represented by the paper.
  4. There is a lack of ablations around the method, including the use of the PCA, architecture of the scaffold decoder, using the visual motion tokens as a target. Additionally, I am unable to easily parse from the tables comparing to the baselines what things added really matter and how VITAL differs from the baselines. A list of how the method differs from the baselines at a high-level would have been helpful. How were the baselines chosen?
  5. The 3D figures, while visually appealing, make understanding a lot harder. It is also not clear where the action decoder in Figure 3 comes from.
Questions for authors
  1. How does the encoder "summarize motion across the visual sequence" in section 3.2?
  2. Why is there a V+ and no V?
  3. What is the temporal and latent regularization for trajectory reconstruction?
  4. Sec. 3.3: h(c) is never defined? At least h isn't.
  5. Does the reward optimization setup follow prior work?
  6. Figure 4: why not compare to other methods too? Otherwise, seems like right it is mostly showing the saturation of the method
4 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer kaCA,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer kaCA,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer kaCA,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Fourth Reminder)

Dear Reviewer kaCA,

As the discussion period draws to a close, we would like to briefly follow up on a point you raised in your review.

We appreciate your pointing out where the terminology made the action representation difficult to follow. In Sections 3.1 and 3.2 of the revised manuscript, we define the visual motion tokens, visual scaffold, and metric residual at first use and use these terms consistently in the text and figures. We also explain how the tokens are decoded into the scaffold and how the residual refines it into the final trajectory. Section 3.2 and the caption of Figure 3 now clarify where the scaffold decoder comes from.

If any part of our response would benefit from clarification, we would be happy to discuss it here. Thank you again for your time and feedback.

Best,

The Authors of Submission 15

Reviewer_LL9SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VITAL, a vision-grounded action learning framework for autonomous driving that addresses the gap between visual motion understanding and precise trajectory generation. Its main contribution is a novel action tokenizer that decomposes trajectories into a visually derived motion scaffold and a normalized metric residual, preserving both visual alignment and fine-grained execution precision. The framework further introduces one-step action generation via flow matching and bounded residual reinforcement learning for efficient trajectory refinement. Extensive experiments on NAVSIM, including reconstruction analysis, controlled ablations, and comparisons with state-of-the-art methods, validate the effectiveness of the proposed design. VITAL achieves 90.8 PDMS on NAVSIM v1, 91.1 EPDMS on NAVSIM v2, and 38.6 EPDMS on navhard with only one denoising step, demonstrating competitive planning accuracy and inference efficiency.

Critical strengths
  1. Novel and well-motivated action representation. The paper identifies an important limitation of existing world-action models: the disconnect between visual motion representations and precise trajectory execution. VITAL addresses this through a principled decomposition into a visual motion scaffold and a normalized metric residual. This is a meaningful contribution because it improves visual-action alignment while preserving the fine-grained precision required for autonomous driving.

  2. Strong empirical performance with remarkable inference efficiency. VITAL achieves competitive results across NAVSIM v1, v2, and navhard using only one denoising step, with a 3.29× inference speedup over ten-step sampling. The controlled ablations further demonstrate the benefits of the proposed representation over raw trajectory prediction and trajectory autoencoding. These results provide convincing evidence of both the effectiveness and practical relevance of the approach.

Critical weaknesses
  1. Insufficient evidence isolating the benefits of the proposed action representation. While the paper demonstrates near-perfect reconstruction using ground-truth residuals, this does not necessarily translate into improved planning when both components must be predicted. The lack of a direct inference-time ablation isolating the predicted residual makes it difficult to determine how much of the performance gain is attributable to the proposed decomposition. Furthermore, the controlled action-learning comparison is conducted under a relatively limited training budget, leaving open whether the observed advantage persists after convergence.

  2. Limited practical gains and robustness concerns regarding residual reinforcement learning. Despite being presented as a major contribution, residual RL provides relatively modest improvements (0.6 PDMS on NAVSIM v1 and 0.1 EPDMS on navhard). More concerningly, it degrades Stage 2 performance on navhard and introduces safety-related regressions in some scenarios, as documented in Appendix C.3. These findings raise questions about whether the proposed refinement strategy delivers sufficiently robust and meaningful improvements to justify its additional complexity.

Questions for authors
  1. Cross-dataset generalization. All experiments are conducted within the NAVSIM benchmark, while the NAVSIM v2 navtest evaluation largely involves rescoring the same scenes. Have the authors evaluated VITAL on any independent driving datasets or benchmarks, such as nuScenes or Bench2Drive? Given that visual-action alignment is a central motivation of this work, cross-dataset validation would be important to establish whether the proposed representation generalizes beyond the NAVSIM setting.

  2. Contribution of the predicted residual. Can the authors provide an inference-time ablation comparing the predicted visual scaffold alone against the complete scaffold-plus-predicted-residual representation? This would help establish whether the residual meaningfully improves planning performance rather than merely increasing reconstruction capacity.

  3. Fairness of the controlled comparisons. Do the improvements over Trajectory AE and direct trajectory prediction persist under fully converged training with matched computational budgets? The current controlled comparison uses only 4,256 optimization steps.

  4. Effectiveness of residual RL. Given the marginal improvement on navhard and the degradation in Stage 2 performance, can the authors clarify under what conditions residual RL provides consistent benefits and how the reported safety-related regressions affect its practical value?

4 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer LL9S,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer LL9S,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer LL9S,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 15

AuthorsOpenReview ↗

Follow-Up to Your Review (Fourth Reminder)

Dear Reviewer LL9S,

As the discussion period draws to a close, we would like to briefly follow up on a point you raised in your review.

We appreciate your suggestion to test whether the predicted residual improves planning when the visual scaffold must also be predicted. In Table 1 of the revised manuscript, we compare a planner that predicts only the visual motion tokens with the full planner that also predicts the residual. Both use one denoising step with the RL actor disabled, so the comparison isolates the residual’s contribution under the same inference settings.

If any part of our response would benefit from clarification, we would be happy to discuss it here. Thank you again for your time and feedback.

Best,

The Authors of Submission 15

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank our reviewers for the constructive feedback!

VITAL's central idea is to define the action expert's learning target through visual motion. Existing dual-expert WAMs supervise video with future scenes but actions with trajectory-only targets. VITAL makes visual motion tokens the principal action target, aligning both experts' supervision around the same recorded motion and linking the action target directly to visual conditioning. A compact metric residual code preserves the precision that vision alone misses. Before any residual RL, this target design reaches 90.2 PDMS versus SimWAM's 66.6 with the same backbone and one denoising step. Residual RL is a lightweight use of this representation.


We appreciate the reviewers' positive feedback:

  • TjAQ and LL9S: addressing the gap between action and video modalities in WAMs.
  • LL9S: a novel, well-motivated representation that improves visual-action alignment while preserving precision.
  • kaCA: reconstruction analysis that motivates the residual representation.
  • LL9S and kaCA: strong one-step performance and inference efficiency relative to baselines.
  • LL9S and TjAQ: convincing ablations and fairly complete experimental analysis.
  • TjAQ: clear figures and project animations.

Our contributions are:

  1. A vision-induced action representation for WAMs. VITAL derives the principal action target from recorded visual motion through a VGGT-Ω\Omega based tokenizer, aligning video and action supervision at target construction. A paired metric residual code, motivated by the reconstruction diagnostic, restores the precision vision misses (123.60 → 0.31 mm ADE with recorded images).
  2. A validated mapping hypothesis with one-step planning. An ideal one-step flow update returns the conditional mean target, so one step succeeds when visual context nearly determines the target. This is supported by visual probes (79.4 vs 52.3 PDMS, matched vs shuffled vision), controlled learning (70.1 / 64.9 / 55.0, VITAL / AE / raw), the visual-only planner (89.96 PDMS), and IL one-step planning (90.2 vs 66.6 for SimWAM).
  3. Representation-supported post-training and generalization. The residual coordinates host a 1.23M-parameter actor with six bounded directions. With one denoising step VITAL reaches 90.8 PDMS at 149 ms (3.29×3.29\times faster than ten steps), 91.1 EPDMS on NAVSIM v2, and 38.6 on navhard, and its IL checkpoint transfers zero-shot to nuScenes and Bench2Drive.

Additional Results & Experiments

Following the reviewers' suggestions, we revised the manuscript. Red marks rewritten text and blue marks additions.

  • Visual-only planner (LL9S, kaCA, TjAQ). Completed before submission; now in Table 1 and Sec. 4.3.1 with all seven metrics. One step without the actor gives 89.96 PDMS for visual tokens alone and 90.2 for the full IL target.
  • Independent datasets (LL9S, TjAQ). nuScenes and Bench2Drive use one NAVSIM-trained checkpoint without fine-tuning or RL (Table 3; full nuScenes comparison in Table A6); PhysicalAI is in Table 4.
  • Theory of one-step planning (kaCA, LL9S, TjAQ). Sec. 3.3 shows that an ideal one-step update returns E[z⋆∣h(c)]\mathbb E[z^\star\mid h(c)], linking one-step quality to the mapping property of vision-induced targets.
  • Stage-wise residual RL analysis (LL9S, TjAQ). Sec. 4.3.4 reports IL and RL results on v1, v2, and every navhard stage, with the scope of the actor explained.
  • Cross-method quality and latency (kaCA). The teaser in Fig. 1 adds a radar chart over NAVSIM v1, v2, navhard stages, and latency.

Other Changes & Modifications

  • Title (TjAQ). Changed to "VITAL: Visually Grounded Action Tokens for One-Step Driving Planning".
  • Abstract, Introduction, and Conclusion (all reviewers). Reorganized around alignment, mapping, the reconstruction diagnostic, and the one-step expectation, with residual RL repositioned as a use of the representation.
  • Terminology and notation (kaCA). Visual motion tokens, visual trajectory scaffold, trajectory residual, metric residual code, V+V^+, cc, and h(c)h(c) are defined at first use in Secs. 3.1 to 3.3.
  • Architecture and training (kaCA, TjAQ). VGGT-Ω\Omega, the decoder origin, temporal and latent regularization, the three training stages, and the trainable modules are described in Secs. 3.1 to 3.3 and 4.1.
  • Ablation locations (kaCA, LL9S). PCA rank, probe, and controlled-budget analyses are summarized in Secs. 4.3.1 to 4.3.3 with pointers to Tables A1, A11, and A13.
  • Related work (TjAQ, kaCA). Added Fast-WAM, ResAD, and PLD, and stated how VITAL differs from SimWAM.
  • Figure captions (kaCA). Fig. 3 explains the decoder, and Fig. 4 states the purpose of each panel.
  • Layout. To keep the main text within ten pages, the qualitative case study moved to Appendix C.1 (Fig. A9) and the complete nuScenes table to Appendix B.3 (Table A6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.