acceptodds
Withdrawn from ICLR 2027

An Action Is Worth One Patch: Unified World–Action Modeling with PatchWAM

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robotic actions and their visual consequences describe the same motion: an action specifies how the robot moves, and a future frame shows where this motion leads, with the robot itself visible in the frame. We hypothesize that a predicted future frame substantially constrains the action, allowing the action to be read out of the predicted motion rather than generated by a second model. Most existing world action models nevertheless attach a separate action expert, which learns the same motion again from robot demonstrations alone. In this work, we introduce PatchWAM (Patch World-Action Model), which treats continuous actions as another type of patch through a fixed, parameter-free mapping called Action-as-Patch, so that the velocity field of a fine-tuned pretrained image generator produces the future frame and the action together. Under a linear probe, the future frame predicted by PatchWAM explains as much of the action’s motion as the ground-truth future frame, and clamping the future frame to a different frame during sampling changes the generated action accordingly. Under matched data and optimization settings, PatchWAM outperforms the dual-expert control at all five RoboTwin 2.0 data scales, with gains ranging from 5.8 to 14.3 percentage points. Under other training protocols, PatchWAM reaches 96.12% on RoboTwin 2.0 with additional augmented demonstrations and 91.8% on LIBERO-Plus.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action model, robot manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_khJDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PatchWAM represents each action step as a patch-width token through a fixed replication-and-padding codec and jointly denoises action and future-image tokens within an image diffusion transformer. It removes a dedicated action expert and evaluates the resulting interface across data scales, image backbones, and manipulation benchmarks. Probes and future-clamping interventions investigate the relationship between predicted images and actions.

Critical strengths
  1. The one-token-per-step interface is simple, executable without a learned decoder, and potentially useful for adapting image backbones. Gains of 5.8-14.3 percentage points over the matched dual-expert configuration across five training-window scales, plus a second-backbone result, constitute meaningful evidence for the complete architecture.

  2. The paper includes informative negative results and qualifications. The learned-linear codec performs almost identically to the fixed codec, and the authors explicitly acknowledge that most success does not depend on direct future-action attention in the tested regime. The probe and clamping experiments provide evidence that generated futures contain additional action-related information and that modifying them can alter generated actions

Critical weaknesses
  1. The conceptual novelty is narrower than the high-level no-action-expert framing suggests. The already-cited Cosmos Policy [1] directly injects normalized, duplicated action values into latent frames without architectural modifications, while Action Images [2] provides another, different route to head-free policy generation. PatchWAM's particular contribution is the per-step single-patch interface within an image backbone, not parameter-free action injection in general. Table 3 supports the sufficiency of the fixed codec, but 94.57% versus 94.50% does not establish a performance advantage of parameter-free encoding. The large dual-expert gains jointly change action processing, attention visibility, and parameter allocation, leaving the source and distinctive benefit of the new interface unresolved.

  2. The future-readout motivation is only partially validated. A future image can constrain motion without uniquely identifying the intervening trajectory, contacts, or gripper timing. The probe explains 29.8% of within-chunk motion variance with the predicted future, versus 12.2% from the current image and 30.1% with the true future: matching the true-future probe is not equivalent to recovering most of the motion. Removing direct future-action attention reduces success by only 0.58 points in a single training run. Clamping establishes sensitivity, not that this pathway explains the main accuracy gain. The authors acknowledge this limitation; my concern is that the remaining engineering contribution is not sufficiently disentangled from a more generic shared-backbone policy benefit.

[1] Kim, Moo Jin, et al. "Cosmos policy: Fine-tuning video models for visuomotor control and planning." arXiv preprint arXiv:2601.16163 (2026). [2] Zhen, Haoyu, et al. "Action Images: End-to-End Policy Learning via Multiview Video Generation." arXiv preprint arXiv:2604.06168 (2026).

Questions for authors
  1. Under the same backbone, data, optimization, and comparable action-processing capacity, what gain remains from the proposed patch interface when attention visibility is matched? A focused factorization is needed, rather than another comparison between complete heterogeneous systems.
  2. Can the future-readout analysis distinguish additional predictive information from a mechanism responsible for improved control, using policy-unseen episodes or repeated-training evidence for the attention ablation? Otherwise, will the authors limit the claim to a useful joint interface with observable but non-dominant future dependence?
  3. Please clarify common evaluation denominators after skipped or replaced simulator runs, and report latency at the actual 10-step evaluation setting if available. These details determine how confidently the practical advantage can be compared.
Reviewer_aMuJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PatchWAM, makes an action as a single token and predicts the action with a single model, without having an action expert. The tokenization is really naive, just copy the action multiple times and zero-pad the rest. With this design, PatchWAM outperforms other strong VLAs (including WAM) in RoboDojo and LIBERO-Plus.

Critical strengths

S1. Method is simple and naive (not overengineered)

  • The tokenization is just repeating the actions and zero-padding the reset, which is even “careless” in some sense.
  • This opens up room for future works and shows the benefit of action tokenization.
Critical weaknesses

W1. No comparison with other existing tokenizers

  • I think we can apply this approach (not having action experts) with existing tokenizers.
  • For example, we can not have an action expert and just use FAST [1] tokenizers instead of the naive patching.

W2. Weak ablation studies

  • It does not have any ablation study on patchifying the actions.
  • For example, whether just using linear embedding could work better (and looks more natural)

[1] Karl Pertsch et al., Fast: Efficient action tokenization for vision-language-action models, Arxiv

Questions for authors

Q1. Comparison with other tokenizers

  • Can you compare this approach with FAST tokenizers?
  • I think this comparison will help showing the contribution of the tokenization method and just using a unified model without the action experts.

Q2. Ablation studies

  • Can you add ablation studies about how to tokenize the actions?
  • It will help understand the importance of tokenization in this approach.
Reviewer_cX4rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PatchWAM, which uses a shared visual diffusion transformer to jointly predict future observations and robot action chunks, without a separate action expert. A fixed, parameter-free Action-as-Patch interface encodes continuous actions and decodes the generated tokens into executable actions. Under matched data and optimization settings, PatchWAM outperforms the dual-expert control across five training-window sampling scales on RoboTwin 2.0.

Critical strengths
  • Consistent improvements in the matched comparisons. PatchWAM outperforms the dual-expert control by 5.8–14.3 percentage points across five sampling scales on RoboTwin 2.0. Additional evaluations on LIBERO-Plus and RoboDojo support its broader applicability, although those comparisons do not isolate architectural effects because training protocols differ.
  • A simple action interface within a shared architecture. The fixed encoding and decoding scheme allows the pretrained visual transformer to process both visual and action tokens without a learned action-specific input/output interface or separate action expert. This is a useful simplification, although it does not improve inference speed.
Critical weaknesses
  • The “action as a readout of the predicted future” contribution needs sharper positioning and stronger support. Shared visual-backbone action prediction with a fixed latent interface already appears in prior work such as Cosmos Policy. Moreover, PatchWAM decodes actions from action tokens jointly denoised with future-image tokens, rather than directly from the predicted frame. The probes and clamping experiments demonstrate that predicted frames contain motion information and can influence actions, but do not establish that this dependence explains the architecture’s performance gains.
  • The matched control does not isolate the benefit of removing the action expert. PatchWAM and the dual-expert control differ in action-specific capacity, token interaction, and inference processing. The results therefore support the effectiveness of the proposed architecture relative to this particular control, but do not determine which change causes the improvement or establish that dedicated action models are generally unnecessary. Furthermore, the success-rate comparisons lack independent training repetitions and uncertainty estimates, leaving the stability of the gains unresolved.
Questions for authors

Beyond representing each action step as a single patch token rather than representing an action chunk as a latent frame, what distinguishes PatchWAM conceptually from Cosmos Policy’s shared-backbone action prediction with fixed latent encoding and decoding? Please clarify which aspect supports the claimed contribution of “actions as a readout of the predicted future.”

Reviewer_nnaPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PatchWAM, a unified world–action model that eliminates the dedicated action expert commonly used in existing WAMs. The proposed Action-as-Patch representation maps continuous robot actions into the latent token space of a pretrained image-generation model through a fixed, parameter-free encoding. A shared diffusion transformer then jointly predicts future-image and action tokens using a flow-matching objective.

The authors investigate the relationship between predicted futures and generated actions through linear probing, clamping interventions, and architectural ablations. Extensive evaluations on RoboTwin 2.0, LIBERO, LIBERO-Plus, and RoboDojo demonstrate competitive performance, while comparisons against a dual-expert baseline show improvements across five training-data scales.

Critical strengths
  1. Strong empirical performance across multiple benchmarks.

PatchWAM achieves competitive results on several challenging robotic manipulation benchmarks, including RoboTwin 2.0, LIBERO-Plus, and RoboDojo. It also consistently outperforms the evaluated dual-expert control across five training-data scales, providing encouraging evidence of the effectiveness of the proposed architecture.

  1. Meaningful analysis of the relationship between future prediction and action generation.

Beyond benchmark evaluations, the paper investigates whether predicted future observations contain action-relevant information and influence action generation. The linear probing experiments show that predicted future frames preserve motion information comparable to ground-truth future frames. More importantly, the clamping interventions demonstrate that modifying future-image tokens can systematically shift generated actions toward the corresponding robot states. These analyses provide useful evidence of a directional relationship between future prediction and action generation.

Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for the proposed action-as-readout mechanism.

The central idea of representing continuous actions directly in the latent space of a pretrained visual generator, without a dedicated action expert, has already been explored by Cosmos Policy. In particular, Cosmos Policy also employs a parameter-free replication-based action encoding and recovers actions through averaging. PatchWAM's primary distinction is representing each action step as a single latent patch rather than encoding an action chunk as a latent frame. While this is a compact design, its methodological advantage over existing approaches is not sufficiently established.

Although the clamping interventions demonstrate that future-image tokens can influence generated actions, the practical benefit of this interaction remains unclear. Notably, the isolated-attention control, which prevents direct interaction between future-image and action tokens, achieves 93.99% success compared with 94.57% for full PatchWAM. If this small difference persists across multiple training seeds, it would suggest that direct future-to-action information transfer is largely unnecessary for successful control, substantially weakening the proposed action-as-readout motivation. Any benefit from future prediction might instead arise from auxiliary supervision through shared parameters, rather than from utilizing the predicted future during action generation. An action-only control would further clarify whether future prediction provides any benefit beyond pretrained visual representations.

  1. Insufficiently controlled comparisons weaken the attribution of performance improvements.

The main controlled experiment in Section 4.4 compares PatchWAM against an ImageWAM-based dual-expert control across five training-data scales. Although the authors match the training data and optimization settings, several important architectural differences complicate the interpretation of the improvements. First, PatchWAM jointly denoises future-image and action tokens during inference, whereas the dual-expert control only denoises actions without generating future frames. Therefore, the comparison does not isolate the benefit of eliminating the dedicated action expert from that of joint future–action inference. Second, the dual-expert control uses a randomly initialized action expert, whereas the original ImageWAM explores initialization from pretrained visual transformer weights. This discrepancy is particularly consequential for the claimed improvements in low-data regimes. Furthermore, the paper does not clearly specify whether the dual expert's training-time attention to future-image tokens is consistent with its action-only inference procedure.

The external benchmark comparisons in Section 4.2 also involve inconsistent training protocols. PatchWAM's headline results on RoboTwin 2.0 and LIBERO-Plus rely on additional augmented demonstrations that are not used by several competing methods. Without these additional data, PatchWAM achieves 94.57% on RoboTwin 2.0, below MotuBrain (95.94%), and 80.2% on LIBERO-Plus, below ImageWAM (85.3%). These differences limit the significance of the reported performance advantages over existing methods.

Questions for authors
  1. Novelty relative to Cosmos Policy. Beyond the more compact action representation, what is the main methodological novelty of PatchWAM compared with Cosmos Policy, which also uses a parameter-free replication-based action encoding within a pretrained visual generator? Could you provide a controlled comparison under the same backbone and training budget to isolate the benefit of Action-as-Patch?

  2. Contribution of future-to-action interaction. The isolated-attention control achieves 93.99% success compared with 94.57% for full PatchWAM, suggesting that direct future-to-action interaction may contribute little to policy performance. Given this small difference, could you report results across multiple independent training seeds to establish whether the improvement is statistically significant?

  3. Dual-expert control and inference consistency. Why does the dual-expert control omit future-image denoising during inference, while PatchWAM performs joint future–action denoising? Could you evaluate a dual-expert variant with joint inference to isolate the effect of eliminating the action expert? Additionally, does the dual expert attend to future-image tokens during training, and if so, how is this handled during action-only inference?

  4. Action-expert initialization. Why is the dual-expert control randomly initialized rather than using the pretrained visual transformer initialization explored by the original ImageWAM? How would this initialization affect the reported improvements across different training-data scales?

  5. Benchmark comparison fairness. Could you clarify the training data and augmentation protocols of the strongest baselines on RoboTwin 2.0 and LIBERO-Plus? Given that PatchWAM's headline results benefit from additional training data, could you provide more comparable evaluations under matched data protocols, or clarify how the reported advantages should be interpreted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.