acceptodds
Under review as a conference paper at ICLR 2027

Do Driving World-Action Models Imagine What They Plan? Image–Action Consistency for Evaluation and Learning

Abstract

Driving world-action models (WAMs) pair a native action with an imagined future. Interpreting these outputs as one coherent prediction requires evidence that they describe compatible motion and respond together to changes in conditioning. We introduce Image-Action Consistency (IAC), which evaluates this relationship through candidate-blind visual measurement and explicit observation coverage. IAC maps video and action into a shared space of yaw and interval longitudinal progress. The Alignment Score (AS) measures within-pair motion agreement, the Response Consistency Score (RCS) checks paired yaw responses to controlled condition changes, and the Grounding Score (GS) measures geometric agreement with independently logged futures. On held-out logged videos, visual progress estimates achieve tolerance accuracy on supported intervals, with coverage. Under the tested DriveWAM intervention protocol, response directions agree in of pairs with detectable responses in both outputs, yet only of all valid pairs produce material action responses. This gap shows why conditional agreement must be interpreted together with response coverage. Paired visual-motion targets also improve PDMS over matched-budget action-only fine-tuning by points on DriveWAM and point on WorldDrive. IAC thus provides both a diagnostic of joint predictions and a source of motion supervision for action learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action models, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4YYzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether the future video and native action produced by a driving world-action model express consistent motion. It proposes an image–action consistency framework with a candidate-blind visual reader, which recovers coarse yaw and longitudinal progress without fitting to the action being evaluated. AS measures agreement within a video–action pair, RCS measures corresponding yaw changes under paired condition interventions, and GS separately compares generated geometry with logged futures. The paper evaluates four driving models and uses detached motion estimates from generated videos as auxiliary targets for action-side fine-tuning. This improves PDMS on DriveWAM and WorldDrive under the reported matched training protocol.

Critical strengths
  1. The problem is meaningful. A generated video can look plausible while expressing motion that differs from the associated action, and making this relationship explicit is useful. The candidate-blind reader, support checks, and coverage reporting are sensible design choices. Separating within-pair agreement, paired response consistency, and reference-based geometry also helps distinguish what each measurement actually establishes. The reader validation and controlled mismatch tests provide useful initial evidence for the evaluation protocol.

  2. The auxiliary learning results are encouraging. Relative to action-only fine-tuning, the paired visual targets improve mean PDMS by approximately 1.54 points on DriveWAM and 1.00 point on WorldDrive over three training seeds. The comparisons with ground-truth-motion and shuffled-target supervision strengthen the evidence that the paired targets provide useful information. PDMS is an independent driving-performance measure, so these results demonstrate practical value beyond simply optimizing the proposed consistency scores.

Critical weaknesses
  1. The validity of the consistency measurements on generated videos needs stronger independent evidence. The logged-video reader validation and controlled perturbations are useful, but generated videos can contain artifacts that affect motion estimation differently. The paper acknowledges this limitation, and candidate-blindness and support gates alone do not resolve it. Furthermore, AS uses coarse yaw/progress channels, and its common-source comparison covers only 30 source identities. A blinded human evaluation of whether the videos express the associated actions, or another independent assessment on generated videos, would help establish whether the reported scores capture the intended notion of consistency across models and artifact types. This would strengthen the paper's central measurement contribution.

  2. The experimental scope does not yet establish the broader decision-making value of consistent imagined videos. The PDMS improvements support the usefulness of the auxiliary training signal, and the paper appropriately avoids claiming that they prove improved video–action consistency. However, these results do not directly show how evaluating consistency helps a planner choose among candidate actions or futures. A targeted candidate-selection experiment, with the same candidate set and matched generation/compute budgets, would provide a more direct test: compare the existing selector against one using consistency filtering or scoring, and assess the selected trajectories with independent driving metrics. Such evidence would make the connection between measuring consistent imagination and making better driving decisions substantially more convincing.

Questions for authors
  1. Is there independent validation of the consistency scores on generated videos, such as a blinded human study or annotations of motion/action agreement? In particular, do the scores track perceived consistency when generated artifacts affect the visual reader? This evidence would materially affect my confidence in the evaluation framework.

  2. Can the consistency measurements improve candidate selection during planning, beyond their use as auxiliary training targets? If results are available, please compare selectors on the same candidates with matched budgets and report independent driving performance. This would materially affect my assessment of the framework's broader utility.

Reviewer_GvyaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Image-Action Consistency (IAC) to evaluate whether the future videos generated by a World-Action Model (WAM) for autonomous driving are consistent with the driving actions simultaneously output by the model. Specifically, the authors propose an Alignment Score, a Consistency Score, and a Grounding Score to assess motion agreement, yaw responses, and geometric agreement, respectively. Experimental results validate the effectiveness of the proposed approach.

Critical strengths

(1) This paper focuses on the consistency between generated future videos and predicted driving actions, which carries significant practical implications.

(2) The authors propose an Alignment Score, Consistency Score, and Grounding Score, which provide a comprehensive evaluation of the task from multiple perspectives; this approach is both innovative and effective.

(3) The authors have verified the effectiveness of their approach through extensive experiments, including various World-Action Models and several controlled evaluations.

Critical weaknesses

(1) Some current evaluation metrics may fail to fully capture the consistency between generated videos and the corresponding actions. For instance, the Alignment Score primarily focuses on the yaw direction; in practical scenarios, however, there may be deviations in the magnitude of a turn, leading to geometric differences between two trajectories, even though the metric might still assign a high score.

(2) The results in Table 2 are derived from videos of real-world recordings. In contrast, synthetically generated videos may exhibit issues such as geometric distortion, object discontinuities, and scene deformations, which can affect depth estimation and camera motion recovery. Consequently, the authors should conduct further, more targeted validation using synthetically generated videos.

(3) The authors utilize paired visual-motion targets extracted from generated futures as auxiliary supervision signals for action learning, and evaluate the resulting performance via PDMS. However, it remains to be demonstrated whether the videos generated by the model actually align more closely with the predicted actions following further optimization.

Questions for authors

The authors should further verify the reliability of IAC on generated videos and whether fine-tuning improves image-action consistency.

Reviewer_K5FVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed IAC to measure the consistency across motion and visual modality predicted by WAMs. To measure the consitency, IAC includes three dimensions: motion alignment, intervention response and grounding score. As the results, IAC is able to capture inconsistent prediction from WAM and help learning better model with such consistency regularization.

Critical strengths
  • Proposed IAC with three evaluation metrics to measure different aspects of visual-action consistency in WAMs
  • Demonstrated meaningful analysis results on existing WAMs for autonomous driving to showcase the necessasity of this three aspects
  • Using the paired IAC target enhanced policy model perfomance on NAVSIM benchmark
Critical weaknesses
  • The proposed IAC analysis is based on a strong assumption where estimated action from predicted video along is genuirely reflecting the intention. The authors didn't provide further evidence on how much error are introduced in this step and how would it affect the final consistency metrics.
  • The baseline for policy training is relatively weaker than the standard RL finetuning with consistency reward. RL finetuning has shown to be effective for improving consistency and RCS can be considered as a special cases of exploration for different future modalities with RL.
  • Limited granularity on consistency measurements. Some nuanced behaviors like gentle steering might not be captured.
Questions for authors
  • While the estimator is validated on logged videos, generation artifacts may violate its geometric assumptions. Can the authors quantify estimation error on generated videos, including cases that pass the support checks, and show how this uncertainty affects AS, RCS, and model comparisons? How can genuine video–action disagreement be distinguished from estimator error?
  • Could the authors compare the proposed auxiliary-supervision approach with RL fine-tuning using a simple consistency reward, under matched data and compute budgets?
  • How sensitive are IAC metrics to gentle steering, small lateral deviations, or mismatches in turn magnitude and timing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.