acceptodds
Withdrawn from ICLR 2027

JAM: Effective Coupling of World and Action in World-Action Model Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

18 views

Abstract

World Action Models (WAMs), which jointly predict future observations and robot actions, have emerged as a promising paradigm for generalist robot control. Their effectiveness is closely tied to how world and action predictions are coupled during pretraining. Existing WAMs condition one modality on the other, merge both into a shared token sequence, or use multi-stream joint attention between world, action, and text. However, these designs restrict information to one direction, let the high-dimensional world signal dominate the compact action signal, or entangle direct world–action coupling with semantic understanding. We introduce JAM (Joint World–Action Modeling), a WAM pretraining framework where world and action streams exchange information and gradients through MM-DiT-style joint attention, while text conditions both streams. JAM further uses 2D motion tracks to provide an embodiment-agnostic image-space target that makes world–action correspondence explicit and extends supervision to action-free human videos. Pretrained primarily on action-labeled robot data, JAM scales consistently with compute and model capacity. Evaluations across seven simulation benchmark families and real-world experiments demonstrate that the same pretrained JAM model adapts to single-arm, bimanual, mobile, and humanoid embodiments and supports action-only inference without generating future observations. Controlled ablations further show that our two-stream world–action coupling outperforms one-way cross-attention and a shared-residual joint-attention baseline, and that motion-track supervision improves downstream performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Robot Foundation Models, World Models for Robot Learning, Joint Action–World Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v7MeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JAM is a framework for pretraining world-action models across embodiments. Its main ingredients are reciprocal computation between separate world and action streams and joint prediction of native robot actions and explicit image-space motion tracks alongside future video. The model uses established MM-DiT-style joint attention, while language conditions both streams through cross-attention. Motion tracks represent displacement from current-frame anchors together with observability, providing a shared visual-motion target for heterogeneous robot data and human videos without executable action labels. Independently corrupted targets and masked objectives accommodate partial supervision. A shared pretrained checkpoint is adapted to seven simulation benchmark families and five real-world Franka tasks. The paper also studies coupling designs, track-label coverage and spatial support, scaling, and future-video K/V dropout for action-only inference.

Critical strengths
  1. Joint prediction of native actions and explicit visual motion is a useful contribution. Motion tracks describe displacement from current-frame anchors together with observability, while native actions retain each robot’s executable control coordinates. Predicting these targets jointly with video provides an explicit supervision interface between controls and their visible effects, and allows human hand-motion data to contribute without inventing robot-action labels. Figures 4B–C show that track-label coverage and spatial support affect downstream success, with end-effector points plus a moderate scene grid giving the strongest reported geometric-control result. This is substantive evidence for the usefulness of motion-track supervision.

  2. Broad practical validation of a shared checkpoint. The model is adapted to single-arm, bimanual, mobile, and humanoid settings and is competitive on several benchmarks. The real-world experiment uses the same 50 demonstrations per task and 20,000 LoRA updates for all compared models, with JAM achieving 71/100 successes versus 52/100 for π0.5 and 49/100 for LingBot-VA (Table 9; Appendix A.5). This supports the practical value of the complete recipe, even though its robustness is not uniformly superior.

Critical weaknesses
  1. The specific cross-embodiment contribution of motion tracks remains insufficiently isolated. Figures 4B–C support the utility of track supervision overall and show that coverage and spatial support matter. However, they do not isolate the additional benefit of human track labels at fixed human-video exposure. Figure 6’s video-prior comparison changes the complete world–action pretraining recipe, so its transfer gains cannot be attributed specifically to the shared motion-track interface. Thus, the evidence establishes the usefulness of geometric supervision more clearly than the claimed role of human tracks in learning transferable world–action correspondence. This matters because bridging native robot controls and action-free human motion is a central contribution of the paper.

  2. The experimental results and protocols are insufficiently auditable. Section 4.4 defines the scaling score as the mean over six benchmarks. The corresponding main-table scores average to 66.22%, whereas Figure 5’s full-run/6.02B endpoint appears around 70–71%. Different checkpoints or adaptation settings could explain this, but the distinction is not documented. More broadly, the manuscript does not specify the exact inherited video checkpoint, complete pretraining budget, per-benchmark adaptation recipes, provenance of most simulation baseline results, or the meaning of the error bars in Figures 4–6. The “held-out embodiment” exclusions are also unclear given RoboCasa365 in the pretraining mixture. These gaps limit assessment of the comparisons, scaling, and transfer claims.

Questions for authors
  1. Motion-track definition: How are robot actuator/scene points and human hand points selected and organized across sources, and how are multiple points packed into the 16 track tokens? Please specify the normalization, displacement gain, observability construction, and coordinate-mask rules, as well as which data sources provide each type of track supervision.

  2. Motion-track attribution: At fixed human-video exposure, what additional cross-embodiment benefit comes from human track supervision? Does jointly generating tracks improve control more than using the same tracks as an auxiliary prediction target under matched data and model budgets? These comparisons would materially clarify the distinct contribution of the proposed interface beyond the existing coverage/form ablations.

  3. Results and protocols: Please reconcile Figure 5’s full-run/6.02B point with the 66.22% mean from Tables 2–5, 7, and 16. Which main simulation baselines were rerun, under what adaptation and rollout settings, and what do the error bars in Figures 4–6 represent? What precisely was excluded from pretraining for Figure 6’s held-out transfer?

Reviewer_ai97OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JAM, a world–action model pretraining framework that jointly predicts future video, native robot actions, and image-space motion tracks. The architecture uses separate world and action streams connected through joint attention, with language conditioning both streams. Motion tracks provide a shared supervision target across embodiments and enable learning from human videos without robot action labels. The authors evaluate a shared pretrained initialization through benchmark-specific adaptation on seven simulation benchmark families and five real-world manipulation tasks.

Critical strengths

The paper addresses a relevant question, i.e., how to couple world prediction and action learning effectively during pretraining. Explicit motion track supervision is a useful component, providing a concrete connection between robot commands and their visible effects while accommodating heterogeneous supervision.

The evaluation covers several embodiments, and the inclusion of real-world experiments strengthens the paper. On the five Franka tasks, JAM achieves better performance than pi05 under the reported adaptation protocol. The controlled architecture and motion-track ablations also provide more direct evidence for the design choices than the cross-model benchmark comparisons alone.

Critical weaknesses
  • The methodological novelty is insufficiently distinguished from prior work. Separate video/action branches, multimodal joint attention, joint flow-matching objectives, and action-only inference already appear in existing frameworks. The paper acknowledges that its attention operator is established, but does not clearly demonstrate what additional principle distinguishes its formulation from close alternatives such as FastWAM, DiT4DiT, and Motus. In particular, preserving cross-stream gradients is not inherently novel in an end-to-end differentiable model, its specific contribution requires stronger isolation.

  • The robustness results weaken the generalization narrative. JAM performs well in several nominal settings but falls substantially behind strong VLA baselines under some perturbations. On LIBERO-Plus, it underperformas pi05. On RoboTwin, its success drops drastically from clean to random, also below pi05. These results suggest that the learned world–action correspondence is not consistently robust to environmental variation. They do not prove memorization, but they leave broad generalization insufficiently established.

  • The cross-model comparisons are difficult to interpret conclusively. The manuscript does not consistently document whether baseline results use matched demonstrations, observation inputs, adaptation budgets, and evaluation configurations. The unusually low FastWAM results particularly warrant clarification. Differences in pretraining exposure also prevent these comparisons from isolating architectural benefits.

  • The evidence has limited statistical and physical scope. Main benchmark tables do not report uncertainty across training seeds, making small improvements difficult to assess. Real-world evaluation covers one robot and five tasks, with only 20 trials per task; it therefore provides limited evidence for broad physical generalization.

Questions for authors
  1. What is the precise methodological advance over FastWAM, DiT4DiT, and Motus? Please distinguish forward information exchange, parameter separation, gradient routing, and auxiliary supervision, and identify which component accounts for the gains.

  2. Can the claimed benefit of reciprocal gradient flow be isolated? A controlled experiment that selectively stops cross-stream gradients while retaining the same forward architecture would help substantiate this claim.

  3. Why does JAM degrade substantially under visual and environmental perturbations? Please provide a breakdown across perturbation types and clarify whether motion-track supervision improves robustness relative to an otherwise matched model without tracks.

  4. How were the baselines obtained and adapted? Please report result provenance, demonstration counts, training settings, observation inputs, and evaluation protocols, particularly for FastWAM’s very low randomized and real-world scores.

  5. How are pretraining and evaluation separated? Please clarify task, scene, trajectory, and embodiment overlap, especially for RoboCasa365 and the held-out embodiment experiments.

  6. Are the reported improvements reproducible? Please provide uncertainty estimates and, where feasible, results across multiple training seeds for the principal comparisons.

Reviewer_ZHcYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces JAM (Joint World–Action Modeling), a framework that jointly learns future world observations and robot actions for generalist robot control. Unlike existing methods, JAM uses a two-stream joint-attention architecture to enable direct information exchange between world and action representations, while using text as a shared condition. It also introduces 2D motion-track supervision to better align world and action predictions and leverage human videos without action labels. Experiments across simulation and real-world tasks show that JAM scales well, generalizes across different robot embodiments, and outperforms alternative world–action coupling designs.

Critical strengths

One strength of JAM is its two-stream joint-attention design, which enables direct information exchange between world and action representations, allowing both to improve each other during pretraining.

Critical weaknesses
  1. Jointly predicting future video observations, robot actions, and other modalities is not new. The main contribution of this paper appears to be a modification of the attention mechanism, which I consider relatively incremental. The novelty of the proposed approach is therefore somewhat limited.

  2. I have a significant concern regarding the RoboTwin results. The paper reports that Fast-WAM achieves 70.2% in the clean setting and 1.2% in the randomized setting. However, the original Fast-WAM paper reports 91.88% and 91.78%, respectively, while the Flex-π paper also reports 91.9% and 91.8%. These discrepancies are substantial and raise concerns about the fairness of the comparison. The authors should clarify the evaluation protocol and provide detailed implementation settings to explain these differences.

  3. The writing and presentation of the paper are somewhat difficult to follow, particularly the explanation and conclusions drawn from Figure 4. For example, the caption of Figure 4(c) states: "C varies Motion Track form and density at fixed labelled exposure; geometric control averages LIBERO-Plus camera and layout success with RoboTwin randomized success." It is unclear what this statement means and how the reported metric is calculated. The authors should provide a clearer explanation of the experimental setup and the conclusions drawn from these ablations.

Questions for authors
  1. Could the authors present the key novelty and contributions of the paper in a more structured manner, clearly distinguishing JAM from existing world–action modeling approaches?

  2. Could the authors explain the significant discrepancy between the reported Fast-WAM results on RoboTwin and those in the original Fast-WAM and Flex-π papers? Please clarify the evaluation protocol, implementation details, and whether the comparison uses consistent settings.

  3. Could the authors clarify the meaning of Figure 4(c), particularly how the "geometric control" metric is computed and why the LIBERO-Plus and RoboTwin results are averaged together?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.