acceptodds
Withdrawn from ICLR 2027

PointAction: 3D Points as Universal Action Representations for Robot Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

World Action Models (WAMs) leverage the broad visual dynamics captured by pretrained video diffusion models, offering a promising path toward generalizable robot manipulation. However, RGB-only rollouts leave 3D motion and fine-grained spatial constraints implicit, making action grounding ambiguous. Meanwhile, scaling action supervision across tasks and embodiments remains costly. We present PointAction, a framework that uses dynamic 3D points as an action interface between video prediction and robot control. PointAction adapts a foundation video model to jointly predict future RGB frames and dynamic 3D pointmaps. This interface expresses proposed robot motion in a common geometric form, which an embodiment-specific diffusion decoder translates into controls. Geometric supervision from videos enables the rollout model to learn beyond paired action data, while target-robot video adaptation and decoder training support transfer with limited demonstrations. Experiments demonstrate strong 4D generation quality on robot scenes, competitive manipulation performance in simulation, and adaptation to two real robot arms unseen during pretraining. These results support 3D point dynamics as a common representation for connecting video prediction to robot control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Video-Action Models, 4D Point Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_H7VmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PointAction, which factorizes video-to-action learning into two stages: a shared joint RGB-XYZ generator and an embodiment-specific point-to-action decoder. The generator is initialized from the LVP image-to-video backbone, encodes RGB and camera-frame XYZ sequences separately with a shared frozen VAE, concatenates the latents along the width dimension, and is post-trained with flow matching on DROID and BridgeData V2 using LoRA. At inference, SAM 3 segments the generated RGB, the corresponding XYZ values at robot-surface pixels are retained, and after farthest-point sampling and PointNet-style encoding a lightweight DiT decodes the full action sequence in one shot for open-loop execution. The evaluation covers three ID/OOD regimes on RoboCasa365, 4D generation quality on held-out DROID and Bridge clips, and two real robot arms unseen during pretraining.

Critical strengths
  1. The real-robot results are the paper's strongest evidence. The two arms differ in hardware, camera, visual domain, and task suite, the demonstration budgets are small, and PointAction clearly outperforms the compared VLA baselines on both, by margins well beyond the sampling noise at those sample sizes. This is a substantive contribution that distinguishes the work from its closest neighbors, and it should not be overshadowed by the small margins in simulation.

  2. The controlled comparison of geometry sources is clean and conservative in direction. It holds the paper's own generated RGB fixed and replaces only the geometry source, and it grants the two post-hoc estimators per-frame oracle re-anchoring while PointAction uses a single first-frame anchor. The conclusion survives this self-disadvantaging setup, supporting one clear and limited claim: predicting geometry jointly with RGB beats estimating it post hoc from the same generated RGB.

Critical weaknesses
  1. The inference latency leaves the method with almost no practical value as a policy model, and the only way to relax it sacrifices closed-loop capability. The serial cost of a single planning pass is on the order of minutes, with the large majority coming from video generation. Per the appendix, the entire trajectory is planned once and executed fully open-loop with no replanning, and the paper itself acknowledges that this precludes reactive control and prevents recovery from slipping objects or contact disturbances. Latency and reactivity here form a trade-off that cannot be satisfied simultaneously: shortening the planning interval makes the cost untenable, while lengthening it extends the open-loop segment and further degrades error correction. The paper reports the cost faithfully but never studies it as a quantity to be optimized — the denoising step count is a single fixed configuration, there is no curve relating steps to success rate, the main results table carries no cost or execution-protocol column, and the strongest baseline is a receding-horizon method operating on the order of seconds.

  2. The action space is never defined anywhere in the paper, which in turn leaves the pipeline from predicted geometry to executable actions unclear. Actions and robot state recur throughout the formalization, the loss functions, the hyperparameter tables, and the architecture figures, yet they are only ever called an action sequence, low-level actions, and robot state, with no physical meaning, dimensionality, coordinate frame, or normalization, and no statement of whether they are joint angles, end-effector poses, or deltas; by contrast every tensor on the geometry side carries an explicit shape, so this is a one-sided omission rather than a matter of writing style. This makes the central claim impossible to assess: whether the mapping from a surface point set without persistent identities to actions is unique differs entirely across three cases — joint angles admit null-space redundancy, end-effector poses require the gripper to be predicted precisely (and the failure-mode analysis names gripper self-occlusion yielding incomplete pointmaps as the leading problem), and deltas accumulate scale error across the open-loop segment. The source of scale is likewise unspecified: depth for part of the data comes from stereo matching and is metric, while another part is monocular estimation with even the intrinsics estimated, yet the two are trained together under a single set of global statistics for normalization. With no code released, a reader can neither implement the method nor judge whether the claimed separation of geometric description from embodiment-specific control commands holds — its content depends entirely on what the separated side actually is.

  3. Neither implicit geometry nor training-time-only geometry is discussed or compared, so the high cost of explicitly predicting geometry is left unjustified. The paper argues from the premise that RGB-only rollouts leave 3D motion implicit, but every discussion of explicit versus implicit serves to motivate its own design, and nowhere does it acknowledge that the implicit route also works: WAM4D[A] explicitly judges that emitting dense 4D incurs expensive decoding and slows action generation, and therefore uses geometry only as a training-time signal while dropping the geometry branch at inference; Flex-pi[B] treats pointmaps as one stream denoised jointly with actions, and its ablation is better controlled than this paper's because it only adds or removes streams without changing the encoder or the sampling budget. The reasons offered for the explicit interface are inspectability and cross-embodiment reuse, but the former is stated as future work with no experiment, and the latter is never tested by freezing the generator and training only the decoder. A more direct comparison is missing as well: the paper already assumes a fixed camera-robot configuration with geometry in the camera frame, so with a known robot model one could register the predicted point cloud and solve inverse kinematics with no learned decoder at all — and this is precisely the comparison on which the paper's distinction from pose-tracking prior work depends. Given a cost gap of two orders of magnitude, without these comparisons a reader cannot tell what producing geometry at inference time buys.

[A] Ying Li, Xiaobao Wei, Jiajun Cao, Hao Wang, Xiaowei Chi, Chengyu Bai, Qianpu Sun, Jiajun Li, Xiaojie Zhang, Peidong Jia, Jian Tang, Sirui Han, Shanghang Zhang. WAM4D: Fast 4D World Action Model via Spatial Register Tokens. arXiv:2606.14048, 2026.

[B] Ge Yan, Jinghao Liu, Yuzhi Fan, Lei Cai, Minwen Liao, Jesse Zhang, Dieter Fox. Flex-pi: A Multi-Stream World-Action Model with Compute Flexibility. arXiv:2608.10860, 2026.

Questions for authors
  1. Could you provide a trade-off curve relating denoising steps to success rate? If the planning cost cannot be substantially reduced, would you agree to add latency and execution-protocol columns to the main results table, stating explicitly that this method plans once and executes fully open-loop while the strongest baseline is receding-horizon? Also, how much physical time does one action chunk correspond to, and what is the effective control frequency?

  2. Please state the physical meaning of the actions (joint angles / end-effector poses / deltas), their dimensionality, coordinate frame, and normalization, the same information for the robot state, and whether these definitions are consistent across the three evaluation settings. In addition, how are stereo-derived metric depth and monocular estimated depth trained together on a common scale, and is a single set of global statistics sufficient to align them? Where does the depth for the first-frame anchor come from at inference (if it comes from sensor depth on the observed first frame, deployment still requires a depth input)?

  3. Could you add two comparisons: (a) against a scheme that predicts geometry only during training and drops the geometry branch at inference, at matched budget, reporting both success rate and latency; and (b) on the same generated geometry, fitting per-frame gripper poses from the predicted robot points and passing them to the same decoder, as well as registering the predicted point cloud to the robot model and solving inverse kinematics with no learned decoder. Could you also report the success rate with the generator frozen and only a new arm's decoder trained, to support the claim that the interface is shareable?

Reviewer_ikeVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PointAction, a framework that uses predicted robot-surface geometry as an intermediate representation between video generation and robot control. Starting from LVP, the authors adapt a video model to jointly predict RGB frames and camera-frame XYZ pointmaps. Robot segmentation extracts per-frame surface point sets, which an embodiment-specific diffusion decoder translates into actions conditioned on the initial robot state. The generator learns from approximately 75K robot trajectories with estimated geometric supervision, without requiring action labels at this stage. Deployment on a new embodiment involves adapting the generator and training a separate action decoder; each predicted 49-step action sequence is executed open-loop.

The main contribution is the explicit geometric interface and its integration with video-based planning. Validation includes RoboCasa365 manipulation under environment and task shifts, adaptation to xArm7 and YAM, decoder-input ablations, and geometry evaluation using both estimated real-world references and simulator ground truth.

Critical strengths

Table 4 directly examines the paper’s central design choice. Robot-only, jointly predicted XYZ achieves 47.7%/44.1% ID/OOD-Env success, compared with 25.1%/20.3% for RGB-only decoding and 28.4%/21.7% when robot geometry is estimated by DA3 from the same generated RGB. The comparison with separately encoded robot-plus-scene points further supports concentrating the action decoder on the predicted embodiment. Table 6 complements these control results with geometry evaluation against simulator ground truth, reducing dependence on estimated reference depth.

On xArm7, PointAction achieves 43.0% average success versus 22.7% for the strongest reported baseline; on YAM, it achieves 43.3% versus 11.7%. These comparisons use the same target-robot demonstration budgets: 50 trajectories per task on xArm7 and 20 on YAM. The gains across both platforms provide meaningful evidence that the interface can support control beyond the embodiments used for joint RGB-XYZ pretraining.

Critical weaknesses

The current pipeline is prohibitively slow for practical robot control. Generating one 49-frame RGB-XYZ rollout takes approximately 372 seconds on a B200 GPU, and SAM 3 segmentation adds another 9 seconds. The resulting 49-step action sequence is executed open-loop, without replanning. Therefore, the method cannot react to grasp failures, contact disturbances, or moving objects during execution. This limitation should be discussed more prominently, and a closed-loop or faster-replanning evaluation would substantially strengthen the practical claim.

The analysis of the 3D point representation is insufficient to support the claimed benefit of predicted future 3D motion. Robot-only XYZ performs best in Table 4, but it is unclear why adding scene points hurts. One possibility is that background XYZ is less accurate because the generator is trained with estimated geometry supervision (stereo/DA3), and noisy generated background points introduce harmful noise into the decoder. Another possibility, as claimed by the authors, is that the generated robot motion already implicitly captures the relevant scene information. The current experiments do not distinguish these explanations. More importantly, the policy improvement may come from access to 3D geometry itself, rather than from predicting future 3D robot motion. The paper should compare against oracle/current-observation XYZ inputs, in particular current-frame GT robot-only XYZ and GT full-scene XYZ, using the same decoder architecture and point budget. This would clarify whether the gain is due to future-motion prediction, reliable 3D geometry, or simply excluding noisy background points.

Questions for authors

Regarding the Critical Weaknesses described above, I would appreciate further explanation.

Reviewer_oc6NOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a WAM model that predicts future RGB and XYZ frames, then uses SAM 3 to segment the robot and extract its point-cloud representation for action prediction.

Critical strengths

None.

Critical weaknesses

I have seen many papers following this pattern this year: evaluating on simple tasks, adding another output modality to a WAM, still collecting finetuning demos, and claiming superiority based on marginal improvements. Nothing is obviously wrong with the paper, but neither the methodology nor the results stand out as particularly compelling.

At least, I encourage to include at least one of the points below.

  • No finetuning for downstream task.
  • 1~2 very challenging, contact-rich tasks where prior methods fail.
Questions for authors

Multi-view consistency: The current predictions seem to rely on wide-view workspace cameras. However, important details often lie in fine-grained contact regions, which may be occluded or occupy only a small number of pixels (e.g., around 100). Reasoning about these interactions may require wrist-mounted cameras. How could the method incorporate wrist-camera observations while maintaining consistency across views? This issue has been around since 2023 when video-prediction as robot policies come out. It is still unresolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.