- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents PointAction, which factorizes video-to-action learning into two stages: a shared joint RGB-XYZ generator and an embodiment-specific point-to-action decoder. The generator is initialized from the LVP image-to-video backbone, encodes RGB and camera-frame XYZ sequences separately with a shared frozen VAE, concatenates the latents along the width dimension, and is post-trained with flow matching on DROID and BridgeData V2 using LoRA. At inference, SAM 3 segments the generated RGB, the corresponding XYZ values at robot-surface pixels are retained, and after farthest-point sampling and PointNet-style encoding a lightweight DiT decodes the full action sequence in one shot for open-loop execution. The evaluation covers three ID/OOD regimes on RoboCasa365, 4D generation quality on held-out DROID and Bridge clips, and two real robot arms unseen during pretraining.
Critical strengths
-
The real-robot results are the paper's strongest evidence. The two arms differ in hardware, camera, visual domain, and task suite, the demonstration budgets are small, and PointAction clearly outperforms the compared VLA baselines on both, by margins well beyond the sampling noise at those sample sizes. This is a substantive contribution that distinguishes the work from its closest neighbors, and it should not be overshadowed by the small margins in simulation.
-
The controlled comparison of geometry sources is clean and conservative in direction. It holds the paper's own generated RGB fixed and replaces only the geometry source, and it grants the two post-hoc estimators per-frame oracle re-anchoring while PointAction uses a single first-frame anchor. The conclusion survives this self-disadvantaging setup, supporting one clear and limited claim: predicting geometry jointly with RGB beats estimating it post hoc from the same generated RGB.
Critical weaknesses
-
The inference latency leaves the method with almost no practical value as a policy model, and the only way to relax it sacrifices closed-loop capability. The serial cost of a single planning pass is on the order of minutes, with the large majority coming from video generation. Per the appendix, the entire trajectory is planned once and executed fully open-loop with no replanning, and the paper itself acknowledges that this precludes reactive control and prevents recovery from slipping objects or contact disturbances. Latency and reactivity here form a trade-off that cannot be satisfied simultaneously: shortening the planning interval makes the cost untenable, while lengthening it extends the open-loop segment and further degrades error correction. The paper reports the cost faithfully but never studies it as a quantity to be optimized — the denoising step count is a single fixed configuration, there is no curve relating steps to success rate, the main results table carries no cost or execution-protocol column, and the strongest baseline is a receding-horizon method operating on the order of seconds.
-
The action space is never defined anywhere in the paper, which in turn leaves the pipeline from predicted geometry to executable actions unclear. Actions and robot state recur throughout the formalization, the loss functions, the hyperparameter tables, and the architecture figures, yet they are only ever called an action sequence, low-level actions, and robot state, with no physical meaning, dimensionality, coordinate frame, or normalization, and no statement of whether they are joint angles, end-effector poses, or deltas; by contrast every tensor on the geometry side carries an explicit shape, so this is a one-sided omission rather than a matter of writing style. This makes the central claim impossible to assess: whether the mapping from a surface point set without persistent identities to actions is unique differs entirely across three cases — joint angles admit null-space redundancy, end-effector poses require the gripper to be predicted precisely (and the failure-mode analysis names gripper self-occlusion yielding incomplete pointmaps as the leading problem), and deltas accumulate scale error across the open-loop segment. The source of scale is likewise unspecified: depth for part of the data comes from stereo matching and is metric, while another part is monocular estimation with even the intrinsics estimated, yet the two are trained together under a single set of global statistics for normalization. With no code released, a reader can neither implement the method nor judge whether the claimed separation of geometric description from embodiment-specific control commands holds — its content depends entirely on what the separated side actually is.
-
Neither implicit geometry nor training-time-only geometry is discussed or compared, so the high cost of explicitly predicting geometry is left unjustified. The paper argues from the premise that RGB-only rollouts leave 3D motion implicit, but every discussion of explicit versus implicit serves to motivate its own design, and nowhere does it acknowledge that the implicit route also works: WAM4D[A] explicitly judges that emitting dense 4D incurs expensive decoding and slows action generation, and therefore uses geometry only as a training-time signal while dropping the geometry branch at inference; Flex-pi[B] treats pointmaps as one stream denoised jointly with actions, and its ablation is better controlled than this paper's because it only adds or removes streams without changing the encoder or the sampling budget. The reasons offered for the explicit interface are inspectability and cross-embodiment reuse, but the former is stated as future work with no experiment, and the latter is never tested by freezing the generator and training only the decoder. A more direct comparison is missing as well: the paper already assumes a fixed camera-robot configuration with geometry in the camera frame, so with a known robot model one could register the predicted point cloud and solve inverse kinematics with no learned decoder at all — and this is precisely the comparison on which the paper's distinction from pose-tracking prior work depends. Given a cost gap of two orders of magnitude, without these comparisons a reader cannot tell what producing geometry at inference time buys.
[A] Ying Li, Xiaobao Wei, Jiajun Cao, Hao Wang, Xiaowei Chi, Chengyu Bai, Qianpu Sun, Jiajun Li, Xiaojie Zhang, Peidong Jia, Jian Tang, Sirui Han, Shanghang Zhang. WAM4D: Fast 4D World Action Model via Spatial Register Tokens. arXiv:2606.14048, 2026.
[B] Ge Yan, Jinghao Liu, Yuzhi Fan, Lei Cai, Minwen Liao, Jesse Zhang, Dieter Fox. Flex-pi: A Multi-Stream World-Action Model with Compute Flexibility. arXiv:2608.10860, 2026.
Questions for authors
-
Could you provide a trade-off curve relating denoising steps to success rate? If the planning cost cannot be substantially reduced, would you agree to add latency and execution-protocol columns to the main results table, stating explicitly that this method plans once and executes fully open-loop while the strongest baseline is receding-horizon? Also, how much physical time does one action chunk correspond to, and what is the effective control frequency?
-
Please state the physical meaning of the actions (joint angles / end-effector poses / deltas), their dimensionality, coordinate frame, and normalization, the same information for the robot state, and whether these definitions are consistent across the three evaluation settings. In addition, how are stereo-derived metric depth and monocular estimated depth trained together on a common scale, and is a single set of global statistics sufficient to align them? Where does the depth for the first-frame anchor come from at inference (if it comes from sensor depth on the observed first frame, deployment still requires a depth input)?
-
Could you add two comparisons: (a) against a scheme that predicts geometry only during training and drops the geometry branch at inference, at matched budget, reporting both success rate and latency; and (b) on the same generated geometry, fitting per-frame gripper poses from the predicted robot points and passing them to the same decoder, as well as registering the predicted point cloud to the robot model and solving inverse kinematics with no learned decoder. Could you also report the success rate with the generator frozen and only a new arm's decoder trained, to support the claim that the interface is shareable?