- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the choice of prediction target in world-action models (WAMs), arguing that dense 3D scene motion should replace future appearance as the supervision signal. Future motion is represented as source-frame-anchored, pixel-aligned 3D motion maps: each source pixel stores the 3D displacement of its corresponding physical point at several future offsets, forming an image-like tensor that is modeled by a pretrained pixel-space image-generation backbone (PixelDiT). A one-way masked attention between the condition branch and the noisy dynamics branch lets the motion loss backpropagate into the condition representation while the condition tokens never read noisy dynamics tokens in the forward pass, so motion denoising can be omitted entirely at deployment and actions are generated directly. Supervision targets are reconstructed on demand from per-frame simulator geometry under a piecewise-rigid scene model, avoiding pre-rendered pairwise flow. Experiments cover RoboTwin 2.0, LIBERO, LIBERO-Plus, and an ARX5 dual-arm robot, together with a prediction-target comparison and a motion-action consistency diagnostic.
Critical strengths
-
The 2D-motion versus 3D-motion pair in the prediction-target comparison is the cleanest evidence in the paper: input modalities, model, and training setup are held fixed, and only the dimensionality of the motion target changes. That comparison, and all other ablations, run under the same full evaluation protocol as the main experiments rather than a reduced subset.
-
The argument for the one-way condition-dynamics attention is constructive: the forward output of the condition tokens depends only on keys and values computed from the condition tokens themselves, so the motion branch can be dropped at inference without changing the input distribution seen by the action branch. The ablation shows the same structure also performs better. The source-frame anchoring is formalized with equal clarity; transporting physical points through rigid-entity poses keeps the supervision well defined after a point becomes occluded or leaves the field of view, which sidesteps the visibility constraint inherent to trajectory labels recovered backward from a target frame.
-
The motion-action discrepancy (MAD) diagnostic operationalizes a question that has rarely been measured: whether the future predicted by the world model agrees with what the policy actually executes. For an architecture whose predictions exist to shape the representation, that disagreement is precisely where the risk lies.
Critical weaknesses
- Several uncited works are closely related and the differences need to be stated. The four works below were all public before submission but are neither cited nor discussed:
- MoMaps [1] introduces a pixel-aligned 3D scene-motion representation called a Motion Map, mapping each pixel of a reference frame to a sequence of 3D coordinates across several time steps, and repurposes an image diffusion model to generate it.
- Pri4R [2] jointly predicts future 3D point trajectories to shape the policy representation, discards the prediction head at deployment, and ablates four supervision targets: 3D point trajectories, depth latents, 2D trajectories, and goal point sets.
- 3D Dynamics-Aware Manipulation [3] introduces three objectives into policy learning (current depth estimation, future RGB-D prediction, and 3D flow prediction) and concludes that they complement one another without sacrificing inference speed.
- JOPAT [4] jointly generates visual latents, 2D point tracks, and actions inside a single denoising DiT, and reports substantial degradation when only the track objective is used.
Of these, MoMaps is the closest in the construction of the motion-map representation, while the other three overlap with this submission on the mechanism of shaping representations through motion supervision without test-time future prediction, and on the conclusion that 3D motion is preferable to other prediction targets. None of this negates the paper's contribution, but framing the effectiveness of dense 3D motion as an open question in the introduction and claiming to establish it in the conclusion is too strong relative to this body of work, and the discussion in Section 2.2 should cover these works as well. I would add the citations with an explicit statement of the differences, and promote the two more specific contributions: the constructive argument for the one-way attention, which I did not find in the works above, and the on-demand reconstruction pipeline for supervision, for which I found no precedent. Separately, Grounding DINO, SAM 2, VGGT-Omega, and SpatialTrackerV2 are used in the main text but have no entries in the bibliography and should be added; PixelDiT has since appeared at CVPR 2026 and the citation can be updated.
- The real-robot experiments lack a target ablation, so it remains unclear whether estimated 3D supervision still provides a positive gain. The central claim that 3D motion outperforms 2D motion and appearance is verified only on RoboTwin, where supervision comes from exact rigid-body poses in the simulator. On the real robot, 3D supervision is instead estimated with VGGT-Omega and SpatialTrackerV2 (Section 4.1), which is a different noise regime altogether, and 2D track estimation is considerably more reliable than 3D displacement estimation. The real-robot setting is therefore exactly where estimation noise is most likely to erase the advantage of 3D over 2D. The experiments currently report only a comparison against four baselines, and those baselines differ from this method in backbone, pretraining data, action representation, and input modalities. Those differences are hard to quantify, so the result can support the claim that this system performs better on a real robot, but not the claim that estimated 3D motion supervision contributed a positive gain. What is needed is to rerun the three prediction-target configurations on the real robot (last-frame RGB, 2D motion, and 3D motion, all supervised through the same estimation pipeline), which is a self-comparison and therefore unaffected by cross-method differences; adding a fourth configuration that removes motion supervision and keeps only the action loss would make it more complete. Given that four baselines have already been evaluated on the real robot, the marginal cost of three or four self-comparison configurations should be modest, and this set of results would directly determine whether the first contribution extrapolates to real data.
References
[1] Jiahui Lei, Kyle Genova, George Kopanas, Noah Snavely, Leonidas Guibas. MoMaps: Semantics-Aware Scene Motion Generation with Motion Maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. arXiv:2510.11107.
[2] Jisoo Kim, Jungbin Cho, Sanghyeok Chu, Ananya Bal, Jinhyung Kim, Gunhee Lee, Sihaeng Lee, Seung Hwan Kim, Bohyung Han, Hyunmin Lee, Laszlo A. Jeni, Seungryong Kim. Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representation. arXiv preprint arXiv:2603.01549, 2026.
[3] Yuxin He, Ruihao Zhang, Xianzu Wu, Zhiyuan Zhang, Cheng Ding, Qiang Nie. 3D Dynamics-Aware Manipulation: Endowing Manipulation Policies with 3D Foresight. arXiv preprint arXiv:2502.10028, 2025.
[4] Jiarui Guan, Wenshuai Zhao, Yue Pei, Ziliang Chen, Arno Solin, Juho Kannala. Point Tracking Improves World Action Models. arXiv preprint arXiv:2605.23856, 2026.
Questions for authors
-
What exactly distinguishes this paper's motion-map representation from the Motion Map of [1] (for instance, storing displacements rather than coordinates, the choice of the source-anchored reference frame, or transporting points through entity poses rather than estimating them from video)?
-
On the real robot, what are the success rates when the supervision target is replaced with 2D trajectories, and when motion supervision is removed entirely? Relatedly, what is the quality of the trajectories estimated by VGGT-Omega and SpatialTrackerV2 (for example, Avg delta or EPE on a subset with existing annotations), and how far is that noise level from simulator ground truth?
4 comments
Response to Reviewer XfMu (2/3)
Q1: differences from MoMaps
We thank the reviewer for the question. As the reviewer notes, MoMaps is the closest representation to ours: both are dense 3D point trajectories aligned with the pixels of a reference frame, a kind of representation long studied as scene flow (Vedula et al., ICCV 1999; RAFT-3D, CVPR 2021) and dense 3D point tracking (SpatialTracker, CVPR 2024; DELTA, ICLR 2025; SpatialTrackerV2). On this common ground, the three points the reviewer lists are exactly the main design differences between the two, and the two also differ in use:
- Storing displacements rather than coordinates. MoMaps stores the 3D coordinates at each time in the reference camera frame; we store displacements relative to the current position of each source point (Eq. 4). Since the current XYZ is already given as input, the displacement target contains only the motion itself: static points have zero displacement, and the prediction concentrates on what moves. This matches the starting point of our paper: what decides a manipulation outcome is often a small displacement of a few points (Sec. 1).
- Source-anchored reference frame. MoMaps is anchored to a single reference image. We fix the camera frame of one view at the source time as the source reference frame and express the current XYZ and future positions of all views in it, so multiple views share one reference frame and no shared simulator coordinate convention is required across platforms (Sec. 3.2). Point identity is anchored at the source frame: each pixel describes the physical point at the source time, not a pixel in a future frame.
- Transporting points through entity poses rather than estimating them from video. MoMaps estimates trajectories from real videos using monocular depth, 3D tracking, and an optimization over occluded intervals. In simulation, we transport each point with the pose of the rigid entity it belongs to (Eq. 11), so the trajectories are exact and remain defined after a point becomes occluded or leaves the field of view; trajectories are built on demand from per-frame geometry, with storage linear in episode length (Sec. 3.4). On the real robot, we use estimated trajectories (see our response to Q2).
- Use. MoMaps generates motion maps as the output of a scene-motion model, targeting semantics-aware scene motion generation and video synthesis. In M-WAM, the motion map is the world target of a WAM: it is trained jointly with actions, shapes the scene features used by the policy, and is not generated at deployment.
Our paper focuses on the fourth point: using this representation as the entire world target of a WAM. Under matched conditions, it outperforms appearance and 2D-motion targets (Table 3), is competitive with or better than appearance-based WAMs (Table 1), and has a lower inference cost (Table 7). In the final version, we will add MoMaps to Sec. 2.2 and state these differences.
Response to Reviewer XfMu (3/3)
W2 and Q2: real-robot target ablation and quality of the estimated trajectories
We agree with the reviewer's assessment: real-robot supervision is estimated and thus noisier than the exact poses available in simulation, and a self-comparison on the real robot is the right way to test whether estimated 3D motion still provides a gain. We have run this experiment as suggested.
Real-robot target ablation. On all three tasks, we compare four configurations: no motion supervision (action-only), last-frame RGB, 2D motion, and 3D motion. All variants use the same demonstrations, inputs, and post-training procedure (motion-loss weight 0.3; no motion loss for action-only), with 20 trials per task. The 3D target is obtained by feeding the sensor depth (with missing regions filled by scale-aligned VGGT-Ω depth) and the camera parameters to SpatialTrackerV2. The 2D target uses 2D tracks from CoTracker3; as the reviewer notes, 2D tracking is generally more reliable than 3D estimation, which gives the 2D variant its most favorable supervision. This makes the comparison stricter for 3D motion than projecting the estimated 3D trajectories to 2D would. The RGB target uses the recorded images directly.
Table R7. Real-robot target ablation (success rate, 20 trials per task).
| Target (real robot) | T1 | T2 | T3 | Avg. |
|---|---|---|---|---|
| None (action-only) | 0% | 5% | 0% | 1.7% |
| Last-frame RGB | 25% | 30% | 30% | 28.3% |
| 2D motion | 20% | 30% | 25% | 25.0% |
| 3D motion (ours) | 40% | 55% | 45% | 46.7% |
3D motion achieves the highest success on all three tasks, averaging 46.7%: 18.3 points above last-frame RGB (28.3%) and 21.7 points above 2D motion (25.0%). Without motion supervision, success drops to 1.7% on average. The 2D variant uses CoTracker3 tracks, which are generally more reliable, whereas the 3D variant uses estimated 3D trajectories; even so, 3D leads 2D by 20–25 points on every task. 2D motion is close to last-frame RGB (25.0% vs. 28.3%). That 3D motion is best and action-only is lowest agrees with the simulation results (Table 3, and Table R2 in our response to Reviewer C3LK), showing that 3D motion supervision still provides a positive gain under real-robot estimation errors.
Tolerance to estimation errors. M-WAM accounts for errors in real-robot geometry on both the input and the supervision side. On the input side, the condition uses sensor depth directly, the input depth is perturbed during training, including errors at depth boundaries, and segmentation is produced automatically by Grounding DINO and SAM 2 rather than taken from ground-truth masks (Sec. 3.3), so the model can tolerate regions where real-robot depth is partially missing. On the supervision side, real-robot post-training starts from the M-WAM model trained on RoboTwin (both settings are dual-arm) and is dominated by the action loss, with the motion-loss weight reduced to 0.3 so that estimation errors do not damage the pretrained model.
Quality of the estimated trajectories. We agree that label quality deserves attention. Since our real-robot data have no ground-truth 3D trajectories, we use the target ablation above to directly test the effect of the estimated supervision on the policy.
We will add these results to the final version.
I thank the authors for the detailed response and for running the real-robot target ablation, which addresses my main concern about whether estimated 3D supervision still provides a gain. I will maintain my score, on the condition that all changes promised in the rebuttal are made in the revision. When adding that ablation, please also report the setup details disclosed in the rebuttal, namely the initialization from the RoboTwin model, the reduced motion-loss weight, and the depth source used for the 3D targets, so that the final version is consistent with the response.
Response to Reviewer XfMu (1/3)
We thank the reviewer for the careful reading and for recognizing the 2D-versus-3D comparison, the constructive argument for the one-way attention, the source-frame anchoring, and the MAD diagnostic.
W1: related work, framing, and contributions
We will cite and discuss all four works in Sec. 2.2. The fundamental difference between our work and these works lies in the question we ask: can dense 3D motion fully replace appearance and become the sole generative world target of a WAM? Table 3 compares it directly with appearance and 2D motion under matched conditions. The differences from each work are as follows.
Framing. In Pri4R, 3D Dynamics-Aware Manipulation, and JOPAT, motion is an auxiliary signal alongside appearance or action prediction, and JOPAT reports a clear degradation when tracks alone serve as the world target. Whether motion can serve on its own as the world target of a WAM is therefore not obvious, and this is precisely the question our paper asks. Our results answer it: under matched conditions, dense 3D motion as the sole world target outperforms appearance and 2D motion, and clearly outperforms an action-only model that predicts no target (93.10% vs. 72.22%; Table R2 in our response to Reviewer C3LK). In the final version, we will pose this question in the introduction in light of these works and state in the conclusion the scope supported by Table 3.
Two contributions. We thank the reviewer for the positive assessment of these two designs, and we will foreground both in the final version as suggested.
Citations. We will add the missing citations for Grounding DINO, SAM 2, VGGT-Ω, and SpatialTrackerV2, and update PixelDiT to its CVPR 2026 version.