acceptodds
Under review as a conference paper at ICLR 2027

M-WAM: Motion-Centric World Action Model

Abstract

World-action models learn robot control alongside future prediction, but their predictive objectives commonly emphasize visual appearance. We investigate dense 3D scene motion as a supervision target for action learning. We introduce M-WAM, which predicts pixel-aligned 3D point trajectories organized into image-like motion maps. A pretrained image-generation backbone models these trajectories, while motion and action objectives jointly optimize the scene features used by the policy. In training, we reconstruct dense trajectories on demand from the per-frame simulator geometry. This avoids precomputing every frame pair and reduces export and storage costs from quadratic to linear in episode length. At deployment, the policy generates actions directly without predicting future motion. In RoboTwin ablations, 3D motion improves success by 10.9 percentage points over last-frame RGB prediction and 6.9 points over 2D motion. M-WAM's mean success rate on LIBERO-Plus is 1.7 points higher than ImageWAM's. Across three real-robot tasks, it achieves twice the mean success rate of LingBot-VA. These results support explicit 3D motion prediction as an effective training objective for manipulation policies. All code and data will be publicly released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action models, robotic manipulation, 3D motion, robot learning, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XfMuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the choice of prediction target in world-action models (WAMs), arguing that dense 3D scene motion should replace future appearance as the supervision signal. Future motion is represented as source-frame-anchored, pixel-aligned 3D motion maps: each source pixel stores the 3D displacement of its corresponding physical point at several future offsets, forming an image-like tensor that is modeled by a pretrained pixel-space image-generation backbone (PixelDiT). A one-way masked attention between the condition branch and the noisy dynamics branch lets the motion loss backpropagate into the condition representation while the condition tokens never read noisy dynamics tokens in the forward pass, so motion denoising can be omitted entirely at deployment and actions are generated directly. Supervision targets are reconstructed on demand from per-frame simulator geometry under a piecewise-rigid scene model, avoiding pre-rendered pairwise flow. Experiments cover RoboTwin 2.0, LIBERO, LIBERO-Plus, and an ARX5 dual-arm robot, together with a prediction-target comparison and a motion-action consistency diagnostic.

Critical strengths
  1. The 2D-motion versus 3D-motion pair in the prediction-target comparison is the cleanest evidence in the paper: input modalities, model, and training setup are held fixed, and only the dimensionality of the motion target changes. That comparison, and all other ablations, run under the same full evaluation protocol as the main experiments rather than a reduced subset.

  2. The argument for the one-way condition-dynamics attention is constructive: the forward output of the condition tokens depends only on keys and values computed from the condition tokens themselves, so the motion branch can be dropped at inference without changing the input distribution seen by the action branch. The ablation shows the same structure also performs better. The source-frame anchoring is formalized with equal clarity; transporting physical points through rigid-entity poses keeps the supervision well defined after a point becomes occluded or leaves the field of view, which sidesteps the visibility constraint inherent to trajectory labels recovered backward from a target frame.

  3. The motion-action discrepancy (MAD) diagnostic operationalizes a question that has rarely been measured: whether the future predicted by the world model agrees with what the policy actually executes. For an architecture whose predictions exist to shape the representation, that disagreement is precisely where the risk lies.

Critical weaknesses
  1. Several uncited works are closely related and the differences need to be stated. The four works below were all public before submission but are neither cited nor discussed:
  • MoMaps [1] introduces a pixel-aligned 3D scene-motion representation called a Motion Map, mapping each pixel of a reference frame to a sequence of 3D coordinates across several time steps, and repurposes an image diffusion model to generate it.
  • Pri4R [2] jointly predicts future 3D point trajectories to shape the policy representation, discards the prediction head at deployment, and ablates four supervision targets: 3D point trajectories, depth latents, 2D trajectories, and goal point sets.
  • 3D Dynamics-Aware Manipulation [3] introduces three objectives into policy learning (current depth estimation, future RGB-D prediction, and 3D flow prediction) and concludes that they complement one another without sacrificing inference speed.
  • JOPAT [4] jointly generates visual latents, 2D point tracks, and actions inside a single denoising DiT, and reports substantial degradation when only the track objective is used.

Of these, MoMaps is the closest in the construction of the motion-map representation, while the other three overlap with this submission on the mechanism of shaping representations through motion supervision without test-time future prediction, and on the conclusion that 3D motion is preferable to other prediction targets. None of this negates the paper's contribution, but framing the effectiveness of dense 3D motion as an open question in the introduction and claiming to establish it in the conclusion is too strong relative to this body of work, and the discussion in Section 2.2 should cover these works as well. I would add the citations with an explicit statement of the differences, and promote the two more specific contributions: the constructive argument for the one-way attention, which I did not find in the works above, and the on-demand reconstruction pipeline for supervision, for which I found no precedent. Separately, Grounding DINO, SAM 2, VGGT-Omega, and SpatialTrackerV2 are used in the main text but have no entries in the bibliography and should be added; PixelDiT has since appeared at CVPR 2026 and the citation can be updated.

  1. The real-robot experiments lack a target ablation, so it remains unclear whether estimated 3D supervision still provides a positive gain. The central claim that 3D motion outperforms 2D motion and appearance is verified only on RoboTwin, where supervision comes from exact rigid-body poses in the simulator. On the real robot, 3D supervision is instead estimated with VGGT-Omega and SpatialTrackerV2 (Section 4.1), which is a different noise regime altogether, and 2D track estimation is considerably more reliable than 3D displacement estimation. The real-robot setting is therefore exactly where estimation noise is most likely to erase the advantage of 3D over 2D. The experiments currently report only a comparison against four baselines, and those baselines differ from this method in backbone, pretraining data, action representation, and input modalities. Those differences are hard to quantify, so the result can support the claim that this system performs better on a real robot, but not the claim that estimated 3D motion supervision contributed a positive gain. What is needed is to rerun the three prediction-target configurations on the real robot (last-frame RGB, 2D motion, and 3D motion, all supervised through the same estimation pipeline), which is a self-comparison and therefore unaffected by cross-method differences; adding a fourth configuration that removes motion supervision and keeps only the action loss would make it more complete. Given that four baselines have already been evaluated on the real robot, the marginal cost of three or four self-comparison configurations should be modest, and this set of results would directly determine whether the first contribution extrapolates to real data.

References

[1] Jiahui Lei, Kyle Genova, George Kopanas, Noah Snavely, Leonidas Guibas. MoMaps: Semantics-Aware Scene Motion Generation with Motion Maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. arXiv:2510.11107.

[2] Jisoo Kim, Jungbin Cho, Sanghyeok Chu, Ananya Bal, Jinhyung Kim, Gunhee Lee, Sihaeng Lee, Seung Hwan Kim, Bohyung Han, Hyunmin Lee, Laszlo A. Jeni, Seungryong Kim. Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation. arXiv preprint arXiv:2603.01549, 2026.

[3] Yuxin He, Ruihao Zhang, Xianzu Wu, Zhiyuan Zhang, Cheng Ding, Qiang Nie. 3D Dynamics-Aware Manipulation: Endowing Manipulation Policies with 3D Foresight. arXiv preprint arXiv:2502.10028, 2025.

[4] Jiarui Guan, Wenshuai Zhao, Yue Pei, Ziliang Chen, Arno Solin, Juho Kannala. Point Tracking Improves World Action Models. arXiv preprint arXiv:2605.23856, 2026.

Questions for authors
  1. What exactly distinguishes this paper's motion-map representation from the Motion Map of [1] (for instance, storing displacements rather than coordinates, the choice of the source-anchored reference frame, or transporting points through entity poses rather than estimating them from video)?

  2. On the real robot, what are the success rates when the supervision target is replaced with 2D trajectories, and when motion supervision is removed entirely? Relatedly, what is the quality of the trajectories estimated by VGGT-Omega and SpatialTrackerV2 (for example, Avg delta or EPE on a subset with existing annotations), and how far is that noise level from simulator ground truth?

4 comments
AuthorsOpenReview ↗

Response to Reviewer XfMu (1/3)

We thank the reviewer for the careful reading and for recognizing the 2D-versus-3D comparison, the constructive argument for the one-way attention, the source-frame anchoring, and the MAD diagnostic.

W1: related work, framing, and contributions

We will cite and discuss all four works in Sec. 2.2. The fundamental difference between our work and these works lies in the question we ask: can dense 3D motion fully replace appearance and become the sole generative world target of a WAM? Table 3 compares it directly with appearance and 2D motion under matched conditions. The differences from each work are as follows.

  • MoMaps uses the same kind of representation as ours, dense pixel-aligned 3D trajectories, generated with an image diffusion model for scene motion generation; it does not learn actions (see our response to Q1 for a detailed comparison).
  • Pri4R regresses per-step 3D displacements of 1,024 sampled points with a lightweight head on a VLA and discards the head at test time. Its target ablation also shows 3D trajectories outperforming 2D trajectories and depth, consistent with our Table 3. Pri4R studies a regression-based auxiliary signal and we study a generative world target, so the two offer complementary support for 3D motion.
  • 3D Dynamics-Aware Manipulation adds regression heads for current depth, future RGB-D, and sparse 3D flow to a policy and removes them at inference, treating 3D flow as one of several complementary auxiliaries.
  • JOPAT jointly denoises visual latents, sparse 2D point tracks, and actions, and generates the tracks at inference; it reports a clear degradation when tracks alone serve as the world target. A likely reason is that 2D tracks provide weaker guidance: they are estimated by a tracker and become unreliable under occlusion, and with RGB-only inputs the model must infer the spatial structure behind the tracks without any geometric prior. Video and 3D trajectory targets both carry such spatial information. Our setting differs from JOPAT in exactly this respect: the inputs of all our variants include XYZ and segmentation, which provide a spatial prior. With these inputs, 2D motion as the sole world target does not degrade but improves over last-frame RGB (86.22% vs. 82.17%), and dense 3D motion gives even stronger guidance (93.10%).

Framing. In Pri4R, 3D Dynamics-Aware Manipulation, and JOPAT, motion is an auxiliary signal alongside appearance or action prediction, and JOPAT reports a clear degradation when tracks alone serve as the world target. Whether motion can serve on its own as the world target of a WAM is therefore not obvious, and this is precisely the question our paper asks. Our results answer it: under matched conditions, dense 3D motion as the sole world target outperforms appearance and 2D motion, and clearly outperforms an action-only model that predicts no target (93.10% vs. 72.22%; Table R2 in our response to Reviewer C3LK). In the final version, we will pose this question in the introduction in light of these works and state in the conclusion the scope supported by Table 3.

Two contributions. We thank the reviewer for the positive assessment of these two designs, and we will foreground both in the final version as suggested.

  • Constructive argument for the one-way condition–dynamics attention. The forward output of the condition tokens depends only on keys and values computed from the condition tokens themselves, so omitting motion denoising at deployment leaves the inputs seen by the action branch unchanged. At the same time, the dynamic tokens read the keys and values of the condition tokens, so the motion loss still backpropagates into the condition representation and shapes the scene features used by the policy (Sec. 3.3). Table 8 also shows that this structure performs better when the world target is dense 3D motion: 3.96 points higher than bidirectional attention, with 36.6% lower latency.
  • On-demand reconstruction of motion supervision. Building the required trajectories on demand from per-frame geometry keeps simulator export and offline storage linear in episode length and keeps every trajectory defined through occlusion.

Citations. We will add the missing citations for Grounding DINO, SAM 2, VGGT-Ω, and SpatialTrackerV2, and update PixelDiT to its CVPR 2026 version.

AuthorsOpenReview ↗

Response to Reviewer XfMu (2/3)

Q1: differences from MoMaps

We thank the reviewer for the question. As the reviewer notes, MoMaps is the closest representation to ours: both are dense 3D point trajectories aligned with the pixels of a reference frame, a kind of representation long studied as scene flow (Vedula et al., ICCV 1999; RAFT-3D, CVPR 2021) and dense 3D point tracking (SpatialTracker, CVPR 2024; DELTA, ICLR 2025; SpatialTrackerV2). On this common ground, the three points the reviewer lists are exactly the main design differences between the two, and the two also differ in use:

  1. Storing displacements rather than coordinates. MoMaps stores the 3D coordinates at each time in the reference camera frame; we store displacements relative to the current position of each source point (Eq. 4). Since the current XYZ is already given as input, the displacement target contains only the motion itself: static points have zero displacement, and the prediction concentrates on what moves. This matches the starting point of our paper: what decides a manipulation outcome is often a small displacement of a few points (Sec. 1).
  2. Source-anchored reference frame. MoMaps is anchored to a single reference image. We fix the camera frame of one view at the source time as the source reference frame and express the current XYZ and future positions of all views in it, so multiple views share one reference frame and no shared simulator coordinate convention is required across platforms (Sec. 3.2). Point identity is anchored at the source frame: each pixel describes the physical point at the source time, not a pixel in a future frame.
  3. Transporting points through entity poses rather than estimating them from video. MoMaps estimates trajectories from real videos using monocular depth, 3D tracking, and an optimization over occluded intervals. In simulation, we transport each point with the pose of the rigid entity it belongs to (Eq. 11), so the trajectories are exact and remain defined after a point becomes occluded or leaves the field of view; trajectories are built on demand from per-frame geometry, with storage linear in episode length (Sec. 3.4). On the real robot, we use estimated trajectories (see our response to Q2).
  4. Use. MoMaps generates motion maps as the output of a scene-motion model, targeting semantics-aware scene motion generation and video synthesis. In M-WAM, the motion map is the world target of a WAM: it is trained jointly with actions, shapes the scene features used by the policy, and is not generated at deployment.

Our paper focuses on the fourth point: using this representation as the entire world target of a WAM. Under matched conditions, it outperforms appearance and 2D-motion targets (Table 3), is competitive with or better than appearance-based WAMs (Table 1), and has a lower inference cost (Table 7). In the final version, we will add MoMaps to Sec. 2.2 and state these differences.

AuthorsOpenReview ↗

Response to Reviewer XfMu (3/3)

W2 and Q2: real-robot target ablation and quality of the estimated trajectories

We agree with the reviewer's assessment: real-robot supervision is estimated and thus noisier than the exact poses available in simulation, and a self-comparison on the real robot is the right way to test whether estimated 3D motion still provides a gain. We have run this experiment as suggested.

Real-robot target ablation. On all three tasks, we compare four configurations: no motion supervision (action-only), last-frame RGB, 2D motion, and 3D motion. All variants use the same demonstrations, inputs, and post-training procedure (motion-loss weight 0.3; no motion loss for action-only), with 20 trials per task. The 3D target is obtained by feeding the sensor depth (with missing regions filled by scale-aligned VGGT-Ω depth) and the camera parameters to SpatialTrackerV2. The 2D target uses 2D tracks from CoTracker3; as the reviewer notes, 2D tracking is generally more reliable than 3D estimation, which gives the 2D variant its most favorable supervision. This makes the comparison stricter for 3D motion than projecting the estimated 3D trajectories to 2D would. The RGB target uses the recorded images directly.

Table R7. Real-robot target ablation (success rate, 20 trials per task).

Target (real robot)T1T2T3Avg.
None (action-only)0%5%0%1.7%
Last-frame RGB25%30%30%28.3%
2D motion20%30%25%25.0%
3D motion (ours)40%55%45%46.7%

3D motion achieves the highest success on all three tasks, averaging 46.7%: 18.3 points above last-frame RGB (28.3%) and 21.7 points above 2D motion (25.0%). Without motion supervision, success drops to 1.7% on average. The 2D variant uses CoTracker3 tracks, which are generally more reliable, whereas the 3D variant uses estimated 3D trajectories; even so, 3D leads 2D by 20–25 points on every task. 2D motion is close to last-frame RGB (25.0% vs. 28.3%). That 3D motion is best and action-only is lowest agrees with the simulation results (Table 3, and Table R2 in our response to Reviewer C3LK), showing that 3D motion supervision still provides a positive gain under real-robot estimation errors.

Tolerance to estimation errors. M-WAM accounts for errors in real-robot geometry on both the input and the supervision side. On the input side, the condition uses sensor depth directly, the input depth is perturbed during training, including errors at depth boundaries, and segmentation is produced automatically by Grounding DINO and SAM 2 rather than taken from ground-truth masks (Sec. 3.3), so the model can tolerate regions where real-robot depth is partially missing. On the supervision side, real-robot post-training starts from the M-WAM model trained on RoboTwin (both settings are dual-arm) and is dominated by the action loss, with the motion-loss weight reduced to 0.3 so that estimation errors do not damage the pretrained model.

Quality of the estimated trajectories. We agree that label quality deserves attention. Since our real-robot data have no ground-truth 3D trajectories, we use the target ablation above to directly test the effect of the estimated supervision on the policy.

We will add these results to the final version.

I thank the authors for the detailed response and for running the real-robot target ablation, which addresses my main concern about whether estimated 3D supervision still provides a gain. I will maintain my score, on the condition that all changes promised in the rebuttal are made in the revision. When adding that ablation, please also report the setup details disclosed in the rebuttal, namely the initialization from the RoboTwin model, the reduced motion-loss weight, and the depth source used for the 3D targets, so that the final version is consistent with the response.

Reviewer_C3LKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

M-WAM jointly learns dense 3D motion prediction and robot control using shared scene features. Pixel-aligned motion maps represent future 3D displacements, while motion generation is omitted at deployment. The method is evaluated on simulation benchmarks and three real-robot tasks.

Critical strengths
  1. Effective motion representation and supervision. The paper proposes pixel-aligned 3D motion maps to incorporate explicit motion supervision into WAM training. Table 3 shows improved control performance over RGB prediction and 2D motion targets, supporting the effectiveness of this design.Table 1 shows that M-WAM offers some performance improvements over the baseline.

  2. A practical separation of training-time prediction and deployment-time control. Motion supervision shapes shared scene features during training. At deployment, the policy directly generates robot action chunks without generating motion maps of future scene trajectories. The combination of competitive manipulation performance in real robot tasks and an 81.39 ms model forward pass on prepared inputs makes this a practically relevant design.

Critical weaknesses
  1. The incremental contribution of generative 3D trajectory supervision is not sufficiently established. The distinctive contribution appears to be the use of dense, source-anchored 3D trajectories as a generative training target for a deployment-decoupled policy. However, future 3D trajectory prediction, trajectory-based supervision of shared representations, and training-only geometric branches have precedents in General Flow, 4D-WAM, and WAM4D. FlowWAM is also directly relevant to the motion-centric formulation but is not discussed. Table 3 supports 3D motion over the tested RGB and 2D targets. However, a same-input action-only baseline is missing. A matched comparison with simpler 3D auxiliary supervision would further clarify the benefit of the generative trajectory objective beyond geometry-conditioned policy learning.
  2. The motion evaluation does not directly establish the claimed 3D modeling advantage. Sections 4.1 and 4.5 specify that EPE and MAD are measured in image-plane pixels, with predicted 3D trajectories projected into 2D. These metrics cannot adequately distinguish depth-direction errors, although motion along the viewing direction is a central motivation for the method. In addition, RGB-based predictions are evaluated through CoTracker3, whereas M-WAM provides trajectories directly; the comparison therefore includes correspondence-estimation error for the RGB baselines. Queries also cover all source-image points, potentially diluting errors on moving objects with static-background points. The policy success results remain valuable, but direct 3D errors and moving/occluded-point breakdowns are needed to substantiate the proposed geometric explanation.
Questions for authors
  1. Controlled evidence request: What is the performance of an action-only model using the same RGB, XYZ, and segmentation inputs, architecture, noise sampling, and training budget? Is there a matched comparison with a simpler geometric auxiliary objective? For Table 3, please clarify whether 2D and 3D motion targets use the same source-point identities, temporal offsets, visibility coverage, and supervision source.

  2. Evaluation request: Can you report metric-space 3D trajectory errors, particularly for moving objects, robot points, and points undergoing occlusion? Please also clarify how much CoTracker3 contributes to the errors attributed to RGB-prediction baselines. These analyses would directly affect my assessment of the claimed 3D advantage.

  3. Efficiency clarification: Table 7 excludes observation preprocessing. What are the end-to-end observation-to-action latency and the costs of constructing XYZ and segmentation inputs? Please specify the input resolution, action horizon and denoising steps used for each method in this comparison.

6 comments
AuthorsOpenReview ↗

Response to Reviewer C3LK (1/4)

We thank the reviewer for recognizing the effectiveness of the motion-map supervision (Table 3) and the practical separation of training-time prediction and deployment-time control.

W1 (first part): relation to General Flow, 4D-WAM, WAM4D, and FlowWAM

The fundamental difference between our work and these works lies in the question we ask. Existing WAMs generally use future appearance as the world target, and recent work adds geometry to it as an auxiliary signal. We ask: does a WAM need to predict appearance at all? Can dense 3D motion serve as the entire world target, so that the model learns how the scene moves rather than how it looks? Visual targets entangle object motion with texture, lighting, and background, whereas the displacement that decides a manipulation outcome may change only a few pixels (Sec. 1).

In each of the three precedents the reviewer identifies, motion or geometry plays a different role:

  • Future 3D trajectory prediction (General Flow): trajectories are predicted by a separately trained model and executed by a heuristic controller at test time; they are not trained jointly with an action policy as a world target.
  • Trajectory-based supervision of shared representations (4D-WAM): the world target remains future video, and trajectory information enters indirectly through feature alignment with a frozen 4D teacher.
  • Training-only geometric branches (WAM4D): the world target remains future video, and geometry is added as an auxiliary future-depth readout.

In these works, appearance remains the world target, or motion is not trained jointly with actions as the world target. We instead drop appearance from the world target and make dense 3D motion the sole generative world target, trained jointly with actions.

FlowWAM (concurrent work) is also motion-centric; the differences lie in the form and use of motion. It generates dense 2D optical flow between adjacent frames in a video latent space and conditions actions on the generated flow at inference. We predict source-anchored 3D trajectories, in which every source point is defined over the whole prediction horizon, including after it becomes occluded, and the motion is not generated at deployment. Whether motion is represented in 2D or 3D is exactly the dimension that Table 3 compares under matched conditions: 3D motion outperforms 2D motion by 6.88 points.

Evidence for this question. We directly compare world targets under identical inputs, model, and training: replacing last-frame RGB with dense 3D motion raises success from 82.17% to 93.10%, and 3D motion also outperforms 2D motion (86.22%; Table 3). We further report the same-input action-only control requested by the reviewer (see Q1 below): with identical geometric inputs, removing the prediction target gives the lowest success rate (72.22%). At the system level, the full model is on par with appearance-based WAMs on RoboTwin 2.0 and LIBERO, better on LIBERO-Plus (84.6% vs. 82.9% for ImageWAM), markedly more robust to camera shifts (86.46% vs. at most 52.64% for the baselines; Table 6), and cheaper at inference (Table 7).

In addition, our second contribution is the on-demand reconstruction of motion supervision from per-frame geometry: simulator export and offline storage grow linearly with episode length, and every trajectory remains defined through occlusion. Reviewer XfMu also notes that they found no precedent for this approach.

In the final version, we will state these differences explicitly in Sec. 2.2 and add a discussion of FlowWAM.

AuthorsOpenReview ↗

Response to Reviewer C3LK (2/4)

W1 (second part) and Q1: action-only baseline, simpler geometric objectives, and matching of the 2D/3D targets

Action-only baseline. We thank the reviewer for suggesting this control. We trained an action-only model whose inputs (RGB, XYZ, segmentation), architecture, initialization, noise sampling, and training budget are identical to those of the variants in Table 3. Only the motion loss is removed, so the condition encoder and the action branch are trained with the action loss alone.

Table R2. Table 3 with the same-input action-only control (RoboTwin 2.0).

Prediction target (same inputs, architecture, noise sampling, and budget)SR (%)
None (action-only)72.22
Last-frame RGB82.17
2D motion86.22
3D motion (ours)93.10

All rows share exactly the same geometric inputs, so the differences between them come from the prediction target rather than from the geometric conditioning itself. Removing the prediction target gives the lowest success rate (72.22%); last-frame RGB, 2D motion, and 3D motion improve over action-only by 9.95, 14.00, and 20.88 points, respectively, with dense 3D motion giving the largest gain. We will add this row to Table 3.

Simpler geometric auxiliary supervision. Our paper addresses what to predict: which world target to use within the generative WAM framework. Table 3 therefore fixes how the target is predicted and varies only the target itself. Replacing the generative objective with a simpler auxiliary form, such as directly regressing 3D displacements from condition features or aligning features with a geometric teacher, is a question of how to predict. Comparing generative modeling, direct regression, and feature alignment on the same 3D motion target is a next step we plan to explore. We also note that the extra cost of the generative objective is incurred only during training: at deployment, motion denoising is omitted entirely, and the dynamic tokens do not take part in inference (Sec. 3.1).

Are the 2D and 3D targets in Table 3 matched? Yes. The 2D target is the projection of the 3D trajectories onto the image plane, so the two share the same source-point identities, the same four temporal offsets (steps 8, 16, 24, and 32 of the 32-step action chunk), the same visibility coverage, and the same supervision source (simulator geometry). Points that later become occluded are supervised in both variants; the 2D target stores the image-plane displacement together with an occlusion flag. The full configuration of Table 3 is given in our response to Reviewer UZjV (Q1).

AuthorsOpenReview ↗

Response to Reviewer C3LK (3/4)

W2 and Q2: motion evaluation

We thank the reviewer for the detailed suggestions. The main evidence for the advantage of the 3D target is the closed-loop success in Table 3: it directly measures task completion and involves neither CoTracker3 nor the choice of query points. Table 4 uses image-plane metrics so that M-WAM can be compared with methods that predict RGB, whose trajectories can only be recovered in 2D by tracking. Following the reviewer's suggestions, we provide three additional analyses below.

(1) Metric 3D trajectory error. The value of depth-direction information for control can be read directly from Table 3. The 2D target is the projection of the 3D target: given the source XYZ, the 3D target contains all the displacement information of the 2D target, and what it adds is exactly the component along the viewing direction. With everything else identical, adding this component raises success from 86.22% to 93.10%. To measure the accuracy of the 3D predictions directly, we compute the endpoint 3D error of M-WAM in XYZ space against simulator geometry, for the three point groups the reviewer mentions. Points that become occluded remain well defined through rigid-entity transport and are therefore included. The evaluation protocol is the same as for Table 4 (Sec. 4.5).

Table R1. Mean endpoint 3D error of M-WAM (cm, ↓) against simulator geometry.

PointsMean 3D error (cm)
Moving objects1.67
Robot1.16
Points undergoing occlusion (objects and robot)1.07

(2) Foreground points only. As stated in Sec. 4.5, the queries in Table 4 cover all source pixels, including the table and background. Following the reviewer's suggestion, we also report Table 4 restricted to foreground points: points on all objects and on the robot (by simulator entity labels), excluding the table and background. Each cell gives all points / foreground points; the all-point values are those of Table 4.

Table R3. Table 4 on all points / foreground points.

MethodPred. Avg. δ (%) ↑Pred. EPE (px) ↓Align. Avg. δ (%) ↑MAD (px) ↓
Fast-WAM91.26 / 79.021.48 / 5.3791.66 / 79.931.26 / 4.20
Motus91.55 / 80.711.67 / 5.5390.65 / 77.141.32 / 4.53
LingBot-VA89.26 / 72.143.85 / 12.2788.19 / 68.825.32 / 17.47
ImageWAMN/A3.51 / 8.71N/A1.76 / 5.91
M-WAM93.06 / 84.461.01 / 3.5091.64 / 81.771.15 / 3.76

Excluding the table and background increases the errors of all methods, which confirms that the all-point metrics are diluted by static points. On foreground points, M-WAM is best on all four metrics. Its prediction EPE is 3.50 px, 34.8% lower than that of Fast-WAM (5.37 px), and its alignment Avg. δ, nearly tied with Fast-WAM on all points (91.64% vs. 91.66%), leads by 1.84 points on foreground points (81.77% vs. 79.93%).

(3) Contribution of CoTracker3. For methods that predict RGB, the errors in Table 4 do include correspondence error introduced by tracking. To quantify this part separately, we run a tracker oracle: CoTracker3 is applied to the ground-truth future frames and compared with the ground-truth simulator trajectories, using the same query points, timestamps, resolution, and scoring as Table 4. The oracle involves no learned prediction, so its error comes entirely from tracking. We evaluate the two input settings used in Table 4: the full video (as for Fast-WAM, Motus, and LingBot-VA) and the source plus final frame (as for ImageWAM).

Table R4. Tracking error of CoTracker3 on ground-truth frames (tracker oracle).

CoTracker3 input (GT frames)Corresponding baselinesAvg. δ (%) ↑, all / foregroundEPE (px) ↓, all / foreground
Full videoFast-WAM, Motus, LingBot-VA93.29 / 89.560.97 / 2.81
Source + final frameImageWAMN/A1.44 / 4.03

Even if a predicted video exactly matched the ground truth, evaluation through CoTracker3 would still incur the errors above. On foreground points, the prediction EPEs of the video baselines (5.37–12.27 px; Table R3) are far above the tracking error (2.81 px), and that of ImageWAM (8.71 px) is likewise far above the tracking error of its setting (4.03 px). Their errors therefore come mainly from the predictions themselves rather than from tracking. M-WAM outputs trajectories directly, without tracking, and its foreground EPE is 3.50 px.

We will add these three analyses to the final version.

Reviewer_UZjVOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

M-WAM investigates whether dense 3D scene motion is a more effective predictive supervision target for robotic action learning than visual appearance or 2D motion and whether it is competitive against SOTA WAMs such as Image-WAM. It represents the future displacements of points visible in the source image as pixel-aligned 3D trajectory maps and jointly trains a PixelDiT-based motion predictor with an action model. Its scene features show motion and action losses shape the scene features used by the policy; asymmetric attention permits direct action generation without future-motion prediction during deployment. A supporting data pipeline reconstructs trajectory targets on demand from per-frame geometry. Experiments cover RoboTwin 2.0, LIBERO, LIBERO-Plus, and three ARX5 tasks. The strongest result is the predictive-target ablation, which reports substantial success gains for 3D motion under stated matched inputs, model, and training settings. The complete method also offers competitive standard-benchmark performance, improved robustness in several settings, and lower model inference latency than ImageWAM.

Critical strengths
  1. A meaningful methodological contribution supported by a substantial target ablation. Table 3 reports 93.10% success for dense 3D motion, versus 86.22% for 2D motion and 82.17% for last-frame RGB: gains of 6.88 and 10.93 percentage points. Section 4.3 states that input modalities, model, and training setup are held fixed. This provides useful evidence about supervision-target design. The contribution of making dense, source-pixel-aligned 3D trajectories the generative future target of a jointly trained action policy is clear.

  2. A useful performance–efficiency tradeoff with downstream corroboration. M-WAM is broadly comparable to ImageWAM on standard RoboTwin and LIBERO averages and reaches 84.6% versus 82.9% on LIBERO-Plus. Table 7 reports 81.39 ms versus ImageWAM's 133.96 ms on an RTX 4090, approximately 39.2% lower model inference latency, excluding observation preprocessing. Across three real tasks, its mean success is 46.7%, versus 23.3% for the strongest baseline on average, LingBot-VA. Although the hardware study is preliminary, these results support practical value beyond the target ablation.

Critical weaknesses
  1. The central comparison needs a more reproducible specification. The supplied material does not sufficiently specify the actual temporal offsets, horizons, backbone configuration and trainable components, training budget, and handling of target-specific heads and loss scales. These details matter because the noise-sampling ablation alone changes success by 21.17 percentage points. The manuscript states that training is controlled in Table 3, so this is a request to verify and reproduce those controls, not evidence that they were violated. The motion-metric standard deviations arise from repeated sampling of fixed models; they do not establish variability across independently trained policies.

  2. The evidence supports a useful target choice more strongly than a broad mechanism claim. The RGB comparison also changes temporal coverage from a final image to a trajectory, and Table 4's projected 2D metrics cannot directly establish depth-direction accuracy. The experiments do not separately demonstrate that density or generative modeling is necessary relative to other geometric supervision. These limitations constrain interpretation rather than invalidate the reported target ablation, and the narrower conclusion that this target improves over the tested alternatives remains valuable.

Questions for authors
  1. Clarification of the central experiment: Please provide or point to the exact Table 3 configurations, especially temporal offsets/horizons, backbone and trainable components, training budget, normalization, and noise sampling. Are 2D and 3D targets temporally matched? How are output-head differences and loss scales handled, and how many independently trained models underlie the success rates?

  2. Clarification of the hardware evidence: How are depth, VGGT-Ω, and SpatialTrackerV2 combined into calibrated metric trajectories, and how are unreliable or occluded tracks handled? Do the compared policies use the same demonstrations, which observation modalities does each receive, and were evaluation initial conditions matched across methods?

  3. Claim scope and an optional diagnostic: Is the intended conclusion that dense 3D motion improves over the tested RGB/2D alternatives, or that generative trajectories are superior to other uses of geometric supervision? The former is supported by Table 3. Metric 3D error on moving points, or an existing matched regression/alignment comparison, would strengthen the broader interpretation if available, but is not an unconditional requirement for accepting the narrower result.

5 comments
AuthorsOpenReview ↗

Response to Reviewer UZjV (1/3)

We thank the reviewer for recognizing the target ablation as a meaningful methodological contribution and the practical performance–efficiency trade-off of M-WAM.

W1 and Q1: configuration of the Table 3 comparison

The three models in Table 3 differ only in the prediction target itself and in the input embedding and output head whose channel count depends on it.

(1) Identical training settings.

  • Temporal range: each action chunk has H = 32 steps; the K = 4 future offsets are steps 8, 16, 24, and 32, so the predicted motion covers the same interval as the actions.
  • Backbone and trainable components: the dynamics branch is initialized from the pretrained PixelDiT text-to-image model; the DiT is fully fine-tuned, and all other pretrained modules are frozen.
  • Training budget: about 100k steps on 8 GPUs, batch size 24 per GPU, learning rate 1e-4, and the same random seed for all three variants.
  • Normalization: inputs and prediction targets are standardized with the training-set mean and standard deviation.
  • Noise sampling: all three variants use the high-noise-biased sampling of Sec. 3.3 (20% of the probability mass on σ ∈ [0.93, 1]), and all use 5 denoising steps at inference. The noise sampling that the reviewer points to is thus identical across the three variants.

(2) The 2D and 3D targets are matched not only in time but point by point. The 2D target is the projection of the same 3D trajectories onto the image plane, so the two share the same source points, the same four offsets, and the same supervision source (simulator geometry). Points that later become occluded are supervised in both variants: for each point and offset, the 2D target stores (u, v, o), the image-plane displacement plus an occlusion flag. The 6.88-point gap between 2D and 3D therefore comes only from the dimensionality of the target.

(3) No bias in output heads or loss scales. For each point and offset, the 3D target stores a 3D displacement and the 2D target stores (u, v, o). Both have the same number of channels (3 per offset, 12 in total), so the output heads have the same size. The RGB target is the last frame of the head and wrist views (3 channels per view). The 2D and 3D targets are standardized with training-set statistics, and the RGB target uses the same image normalization as the RGB inputs. All channels, including the occlusion flag, are trained with the same flow-matching MSE, and every variant simply sums the target loss and the action loss (Sec. 3.3), with no variant-specific weighting.

Number of independently trained models. Each success rate comes from one trained model, evaluated with the same full protocol as the main experiments: 50 tasks × 2 conditions (Clean / Randomized) × 100 trials, i.e., 10,000 episodes (Sec. 4.1). The gaps of 10.93 and 6.88 points in Table 3 are both measured at this scale. As stated in Sec. 4.1, the ± values of the motion metrics reflect repeated sampling of the same model with different noise initializations; they describe sampling variability rather than variability across training runs, and we will make this distinction explicit in the final version.

We will add the complete configuration above to the appendix.

AuthorsOpenReview ↗

Response to Reviewer UZjV (2/3)

W2 (first part): temporal coverage of the RGB comparison

We thank the reviewer for pointing this out. Last-frame RGB predicts a single future time, whereas the motion targets predict four, so their temporal coverage indeed differs.

First, on our backbone, extending the RGB target to four future frames would not give a comparison on equal footing with the motion target. PixelDiT is an image-generation backbone with spatial attention only and no attention along time. The motion map can represent multiple times on such a backbone because it is source-anchored: each source pixel stores, along the channel dimension, the displacements of the same physical point at the K offsets (3K channels, Eq. 5). The noisy motion map is therefore pixel-aligned across offsets: the channels at each location always describe the same physical point. Future RGB frames lack this alignment: as objects and the gripper move, the same pixel location corresponds to different physical points in different frames. Stacking several RGB frames along the channel dimension would mix misaligned content at the same location, and the backbone has no temporal attention to establish correspondences across frames; tiling the frames spatially would instead multiply the number of tokens. On the same backbone, the comparison with matched temporal coverage is 2D versus 3D: both are source-anchored with identical source points and offsets, and the 6.88-point gap comes only from the dimensionality of the target.

Second, multi-frame appearance prediction requires a video backbone that can model correspondences across frames, which is exactly the approach of Fast-WAM. Fast-WAM builds on the Wan2.2-5B video-generation model (6B parameters in total) and is supervised with future video; M-WAM has about 2.6B parameters, its backbone is pretrained only on images, and it is conditioned on geometry and supervised with 3D motion. M-WAM achieves higher average success than Fast-WAM on every benchmark (RoboTwin 2.0: 93.10 vs. 91.83; LIBERO: 98.3 vs. 97.6; LIBERO-Plus: 84.6 vs. 51.4; real robot: 46.7 vs. 21.7) with about a quarter of the inference compute (673.84 vs. 2849.30 GFLOPs) and about a third of the model forward latency (81.39 vs. 238.00 ms). This indicates that, with geometric conditioning and geometric supervision, a smaller image backbone outperforms a video-supervised WAM in both effectiveness and efficiency.

We will add this explanation to Sec. 4.3 in the final version.

W2 (second part) and Q3: depth-direction accuracy and metric 3D error

Projected pixel metrics indeed cannot measure error along the viewing direction. Table 4 uses image-plane metrics so that M-WAM can be compared with methods that predict RGB, whose trajectories can only be recovered in 2D by tracking.

Whether depth-direction information matters for control is already shown directly by Table 3. The 2D target is the projection of the 3D target: given the source XYZ, the 3D target contains all the displacement information of the 2D target, and what it adds is exactly the component along the viewing direction. With everything else identical, adding this component raises closed-loop success from 86.22% to 93.10%.

To measure the accuracy of the 3D predictions directly, we compute the endpoint 3D error of M-WAM in XYZ space against simulator geometry, for points on moving objects, points on the robot, and points that undergo occlusion (on objects and the robot). Occluded points remain well defined through rigid-entity transport and are therefore included. The evaluation protocol is the same as for Table 4 (Sec. 4.5), except that errors are computed in XYZ space rather than on the image plane.

Table R1. Mean endpoint 3D error of M-WAM (cm, ↓) against simulator geometry.

PointsMean 3D error (cm)
Moving objects1.67
Robot1.16
Points undergoing occlusion (objects and robot)1.07

We will add this analysis to the final version.

AuthorsOpenReview ↗

Response to Reviewer UZjV (3/3)

W2 (third part) and Q3: scope of the claim

We thank the reviewer for the comments. Our intended conclusion is the former: under identical inputs, model, and training, dense 3D motion is a more effective prediction target than last-frame RGB and 2D motion. This also follows from our study design: Table 3 fixes how the target is predicted (the same generative framework and the same training) and varies only what is predicted. We do not claim that generative trajectory modeling is superior to all other forms of geometric supervision, and we will state this scope explicitly in the introduction and conclusion of the final version.

The question the reviewer raises, namely which properties of the target matter most, is a valuable one. Our study focuses on the choice of target representation; separating the roles of density and generative modeling is a natural next step. In particular, whether sparser trajectories on a small set of key points already suffice is a direction we plan to explore.

Q2: real-robot evidence

On the real robot, depth is used in two places: the input condition and the output supervision.

Input condition. The model's XYZ condition is obtained by directly back-projecting the depth measured by the sensor, without completion. During training, we perturb the input depth (Sec. 3.3), including errors at depth boundaries, so the model can tolerate regions where real-robot depth is partially missing.

Output supervision. The metric trajectories are constructed in three steps. (1) A stereo depth sensor provides a metric depth map and a validity mask. (2) VGGT-Ω estimates depth up to scale; we compute a scale factor that aligns it with the valid sensor depth and use the aligned estimate to smoothly fill the invalid regions of the sensor depth. (3) The completed RGB-D frames and the camera parameters are fed to SpatialTrackerV2, whose 3D point trajectories serve as the motion supervision. Metric scale comes from the sensor depth and the camera parameters. We do not filter trajectories by confidence or occlusion: all estimated trajectories are used for training, consistent with the simulation setting, in which occluded points are also supervised. Real-robot post-training starts from the M-WAM model trained on RoboTwin and is dominated by the action loss: the weight of the motion loss is reduced to 0.3 so that estimation errors do not damage the pretrained model.

Comparison conditions. All methods use the same 200 demonstrations per task and the same three cameras (one head camera and two wrist cameras). The baselines use their official input configurations; M-WAM receives RGB, XYZ, segmentation, robot state, and language. Object placements are randomized in every trial, with the same randomization procedure for all methods; trials are not paired one-to-one across methods.

In addition, in our response to Reviewer XfMu we report a real-robot target ablation (Table R7): with estimated supervision, 3D motion reaches 46.7%, compared with 28.3% for last-frame RGB, 25.0% for 2D motion, and 1.7% for action-only.

We will add the complete pipeline to the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the three reviewers for their careful and constructive comments. We are glad that the reviewers found the comparison of prediction targets in Table 3, under matched inputs, model, and training, to be substantive evidence (UZjV, C3LK, XfMu); the design in which motion supervision shapes the scene features during training while actions are generated directly at deployment to be practical and well argued (C3LK, XfMu), with a good balance of performance and efficiency (UZjV, C3LK); and the source-anchored formulation, the on-demand supervision, and the motion–action discrepancy (MAD) diagnostic to be clear and useful (XfMu).

The question our paper answers. Existing WAMs generally use future appearance as the world target, and recent work adds geometry to it as an auxiliary signal; JOPAT further reports a clear degradation when tracks alone serve as the world target. Our paper asks: does a WAM need to predict appearance at all? Can dense 3D motion serve as the entire world target, so that the model learns how the scene moves rather than how it looks? With the results added in this response, the evidence is as follows. Under identical inputs, model, and training, an action-only model that predicts no target reaches 72.22%, last-frame RGB 82.17%, 2D motion 86.22%, and dense 3D motion 93.10% (Table 3, Table R2). With estimated supervision on the real robot, 3D motion is still the best and action-only still the lowest (Table R7). At the system level, M-WAM is competitive with or better than appearance-based WAMs at a lower inference cost.

New results in this response.

  • Table R1 (UZjV, C3LK): endpoint 3D error computed directly in XYZ space: 1.67 cm on moving objects, 1.16 cm on the robot, and 1.07 cm on points undergoing occlusion.
  • Table R2 (C3LK): the same-input action-only control reaches 72.22%, below every prediction target; dense 3D motion is 20.88 points higher.
  • Table R3 (C3LK): Table 4 restricted to foreground points (excluding the table and background); M-WAM is best on all four metrics.
  • Table R4 (C3LK): tracking error of CoTracker3 on ground-truth frames (tracker oracle); on foreground points, the prediction errors of the baselines are far above the tracking error and come mainly from the predictions themselves.
  • Tables R5 and R6 (C3LK): inference settings and end-to-end latency; including XYZ construction and segmentation, M-WAM takes 102.99 ms, still below ImageWAM (133.96 ms) and Fast-WAM (238.00 ms).
  • Table R7 (XfMu): real-robot target ablation: 3D motion 46.7%, 2D motion 25.0%, last-frame RGB 28.3%, action-only 1.7%; under estimated supervision, 3D still leads 2D by 20–25 points on every task.

Planned changes for the final version. We will make the following changes:

  • Cite and discuss MoMaps, Pri4R, 3D Dynamics-Aware Manipulation, JOPAT, and FlowWAM (concurrent work) in Sec. 2.2, and state the differences from General Flow, 4D-WAM, and WAM4D (C3LK, XfMu).
  • Pose our question in the introduction in light of these works, and state in the conclusion the scope supported by Table 3 (UZjV, XfMu).
  • Foreground the constructive argument for the one-way condition–dynamics attention and the on-demand reconstruction of motion supervision (XfMu).
  • Add the complete training configuration of Table 3 and the real-robot supervision pipeline to the appendix, explain the temporal coverage of the RGB target in Sec. 4.3, and distinguish sampling variability from variability across training runs in the motion metrics (UZjV).
  • Add all new results above, add the citations for Grounding DINO, SAM 2, VGGT-Ω, and SpatialTrackerV2, and update PixelDiT to its CVPR 2026 version (XfMu).

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.