acceptodds
Under review as a conference paper at ICLR 2027

CtrlWAM: Controllable World Action Models with Aligned Intent and Foresight

11 views

Abstract

World action models (WAMs) jointly predict actions (intent) and visual future (foresight). Standard training adds noise to recorded actions and video simultaneously, but such training paradigms introduce a mismatch: perturbed actions imply counterfactual future visual, while the noised video remains tied to the GT recording. In low-noise regime, the scene geometry and even the dynamic behavior remain clearly visible from the noisy future frames despite the added noise. We present CtrlWAM, which executes perturbed actions in a simulator and pairs them with their noised visual consequences for joint WAM learning. To accommodate the different denoising requirements of video and actions, we introduce warped video–action noise schedules that aim to keep visual layout responsive as action predictions evolve. We further extend the action interface from ego-only control to a variable number of agent streams, allowing a unified model to represent predicted or commanded futures for multiple agents. Driving experiments show more accurate action forecasts, closer agreement between generated video and actions, and better following of supplied commands; robotics experiments show stronger motion fidelity and controllability. Matched controls support the benefit of off-path renders for command following and manipulation fidelity. Together, these findings contribute to a more controllable world action model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_h6FxOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CtrlWAM, a unified world action model designed to improve the physical consistency between predicted actions (intent) and generated visual futures (foresight). The central observation is that conventional joint video-action diffusion independently perturbs recorded actions and videos, potentially producing intermediate denoising states in which the perturbed actions are inconsistent with the visual observations.To address this issue, the authors introduce three components:physically aligned noising,warped video-action noise schedules and structured multi-agent conditioning.The method is evaluated on autonomous driving and robotic manipulation. On the driving benchmark, CtrlWAM-M reduces Self ADE from 0.538 m to 0.513 m, Rollout ADE from 2.920 m to 2.804 m, and Cmd ADE from 1.023 m to 0.981 m compared with the SFT baseline. On RoboTwin/WorldArena, the proposed method improves EWMScore from 58.70 to 61.75 over Cosmos 3 SFT.

Critical strengths

1.This paper identifies an important issue in joint video-action generation: independently noised actions and visual observations may not correspond to the same physical future. This issue is particularly relevant for embodied applications in which action trajectories and visual consequences must remain consistent. The motivation is intuitive and clearly illustrated in Figures 1-3.

2.Physically aligned noising provides a concrete way to incorporate the visual consequences of action perturbations into model training. Rather than introducing only an additional consistency loss, the method modifies the training inputs to expose the model to counterfactual action-observation pairs. This is a meaningful training-design contribution.

3.The experiments cover autonomous driving and bimanual robotic manipulation, including action prediction, video-action consistency, command following, and video generation quality. The paper also provides comparisons across rendering sources and noise schedules, as well as matched-budget robotics controls.

Critical weaknesses
  1. Table 2 reports a VLM compliance improvement from 10.07% for CtrlWAM-S to 18.94% for CtrlWAM-M. However, Appendix E.2 explicitly states that only CtrlWAM-M receives the edited surrounding-agent command, whereas the other baselines do not receive this information.Therefore, the comparison primarily demonstrates the advantage of providing an explicit agent-control interface over models without such an interface. It does not fully isolate the effectiveness of the proposed representation or conditioning mechanism.

  2. An absolute compliance score of 18.94% remains relatively low, raising questions about the practical reliability of counterfactual agent control.A stronger comparison would include an alternative agent-conditioned baseline receiving the same command information, along with quantitative trajectory-following metrics for controlled agents.

  3. The ablation results in Table 4 are informative but reveal a nuanced relationship between the proposed components. The GT-Src shared-schedule configuration achieves a lower Rollout ADE (2.968 m) than the uncleaned Rendering warped configuration (3.019 m), while the Cleaned warped configuration achieves the best Rollout ADE (2.881 m).This suggests that rendering quality and the proposed noise schedule may interact substantially. Furthermore, the warped schedule decreases PSNR for all three rendering sources in Table 4, indicating a trade-off between motion-related metrics and visual reconstruction quality.

  4. The method requires simulator execution and rendering of perturbed trajectories. The driving training set contains approximately 269,000 aligned items, and the experiments rely on a large pretrained backbone and substantial GPU resources.This paper does not sufficiently quantify the additional simulation and rendering costs, preprocessing time, or training-efficiency trade-offs relative to standard fine-tuning.

Questions for authors

Please refer to the weakness

Reviewer_K4f9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CtrlWAM to address physical inconsistencies between action predictions and visual futures in joint video–action generation. Specifically, the authors introduce Physically Aligned Noising, which executes perturbed actions in a simulator and pairs them with their corresponding rendered videos to construct physically consistent training inputs. The paper further proposes Warped Video–Action Noise Schedules to address mismatched denoising dynamics between video and action modalities, and extends the framework with structured multi-agent representations for prediction and control. Experiments on autonomous driving and robotic manipulation demonstrate the effectiveness of the proposed approach.

Critical strengths

The paper identifies physical inconsistencies between video and action during joint denoising, supported by observations of their different convergence behaviors.

Aligned noising and warped noise schedules improve video–action consistency, while the unified framework supports flexible prediction and control.

Experiments on driving and robotics demonstrate improvements in prediction and controllability, supported by relevant ablation studies.

Critical weaknesses

Unclear benefit of aligned noising for counterfactual control. Aligned noising trains the model to recover the original trajectory from physically consistent but perturbed action–video pairs. For example, it may learn to recover a right-turn trajectory from a left-turn action and video, while inference requires following the specified left-turn action. It remains unclear how this training strategy improves counterfactual action following.

Limited justification for the alignment direction. Aligned noising only aligns video with perturbed actions, without exploring whether the reverse direction could benefit inverse dynamics prediction.

Insufficient validation of warped noise scheduling. The paper demonstrates improvements in final performance but lacks direct analysis of video–action consistency throughout the denoising process.

Questions for authors

Why is the original recorded trajectory preferred over the simulated counterfactual trajectory as the supervision target? In aligned noising, the model learns to recover the recorded future from physically consistent but perturbed action–video pairs. Could the authors clarify why this corrective training strategy improves counterfactual action following? An ablation that directly supervises on simulated counterfactual trajectories under matched data and training budgets would help justify this design choice.

Why is aligned noising designed to align video with perturbed actions rather than the reverse? Given that the model is trained on both forward and inverse dynamics tasks, could the authors clarify this design choice and investigate whether video-driven alignment would benefit inverse dynamics prediction?

Does warped noise scheduling effectively improve video–action consistency during denoising? Figure 3 shows that video-implied motion stabilizes earlier than predicted actions. Could the authors quantitatively compare the discrepancy between video-implied motion and predicted actions across denoising steps, with and without warped scheduling?

Reviewer_UEvQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CtrlWAM changes the corruption process of a joint video–action model. Perturbed commands are executed in a simulator, and the executed actions and rendered observations form physically corresponding training inputs; the targets remain the original recorded future actions and video. A warped schedule keeps video noisier than actions, and additional action streams enable prediction or conditioning for selected surrounding agents. Experiments use a Cosmos 3-Nano backbone on reconstructed driving scenes and RoboTwin manipulation. The approach makes a substantive change to joint denoising, with promising matched-budget offline manipulation results. My main concerns concern the sampling interpretation and the strength of the driving/control evidence.

Critical strengths

Physically aligned corruption is an interesting intervention in joint modeling. The method constructs simulator-grounded input pairs and explicitly distinguishes counterfactual inputs from recorded clean targets. Modality-specific timesteps and flexible conditioning have precedent in Unified World Models; the more distinctive contribution is coupling perturbed actions to their rendered visual consequences during corruption. This is a substantive modeling choice beyond adding a conditioning interface (Sections 3.1–3.2).

The matched manipulation results provide useful evidence for the combined recipe. All arms start from a common 2,000-iteration checkpoint and receive 2,000 additional iterations. GT-Src retains perturbed actions and the aligned recipe while replacing renders with recorded frames. CtrlWAM improves EWMScore from 58.70 for continued SFT and 58.65 for GT-Src to 61.75; trajectory accuracy improves from 0.302 for SFT to 0.412. The full metric breakdown also makes clear that image and aesthetic quality do not improve. The driving ablation shows an interaction between render source and schedule, supporting the combined recipe rather than independent benefits from every component (Tables 3–4 and 7; Appendices B–D).

Critical weaknesses
  1. The endpoint-regression identity does not by itself establish the generative interpretation of the modified corruption process. Appendix A correctly shows that the squared-noise-weighted target f∗=(xσ−xGT)/σf^* = (x_\sigma-x_{\mathrm{GT}})/\sigma is equivalent to clean-endpoint regression. After simulator execution and rendering, however, xσx_\sigma generally does not follow a fixed Gaussian interpolation to the recorded endpoint. Its secant toward that endpoint need not be the derivative of the simulator-induced path. Appendix A.3 explains how data-prediction UniPC and paired modality grids are implemented, but does not establish their compatibility with the altered training distribution or the sampling prior. Recorded-data anchors provide important standard-noising coverage, but do not alone resolve that interpretation. This is not evidence that endpoint regression cannot work; the paper should clarify the intended probability path and prior, or explicitly present the solver as a heuristic supported by targeted training–sampling distribution diagnostics (Section 3.1; Appendix A).
  2. The driving and selected-agent results need stronger support for the claimed controllability gains. Against matched SFT, CtrlWAM-M changes Self, Rollout, and Cmd ADE from 0.538/2.920/1.023 m to 0.513/2.804/0.981 m, respectively, without paired uncertainty estimates. Self and Cmd ADE rely on an inverse dynamics model (IDM); its real-video reference of 0.523 m ADE makes calibration relevant, but is neither a statistical noise floor nor evidence that the gains are insignificant. Separately, the selected-agent comparison gives edited agent commands only to CtrlWAM-M, so it does not isolate aligned corruption from access to the command. The reported 18.94% is a mean Gemini compliance score, not an episode success rate, and lacks independent calibration. Together these issues limit how confidently the results establish stable control improvements (Tables 1–2; Appendices B.3 and E). I currently lean toward rejection. Clarifying or diagnosing the sampling interpretation, establishing the stability of the driving improvements, and validating selected-agent control against a baseline with the same command interface could move my recommendation toward acceptance. These requests concern the paper’s existing offline/open-loop claims; they do not require a new real-robot or closed-loop benchmark.
Questions for authors

Clarification and targeted diagnostics: What probability path and sampling prior justify the aligned endpoint target? If the solver is intentionally heuristic, can you state that scope and provide evidence about prior/noise-level coverage or sensitivity to the anchor mixture? Targeted validation: Can you report paired confidence intervals over the existing driving evaluation clips, and a focused independent check of IDM accuracy on generated and counterfactual videos? Attribution and calibration: How does a multi-agent model with the same command interface but standard corruption perform? Does a small agent-identity-matched trajectory or human evaluation agree with the VLM compliance scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.