acceptodds
Under review as a conference paper at ICLR 2027

PointPilot: Unified Robotic Manipulation via Object-centric 3D Point Flow Prediction

Abstract

Generalization remains a central challenge in robotics, calling for policies that address domain variations from multiple dimensions such as diverse embodiments, camera poses, and/or scene layouts. In order to approach this problem, we introduce PointPilot, a task-focused and 3D aligned manipulation framework built on a 3D space point-flow prediction. Rather than modeling the global scene, PointPilot only represents task-focused objects (robot grippers treated as objects), avoiding the prediction of potentially redundant scene parts unrelated to the ma- nipulation tasks. Moreover, PointPilot aligns 3D spatial representation in one unified coordinate and supports closed-loop and fully automatic action execution across different manipulation modes from object point flow without a separately trained action expert, while existing works typically need an extra action head or are (semi-)automatic and single-arm mode specific. Given sensor observations and lan- guage instructions, our model jointly learns subtask planning, task-relevant object grounding, and 3D point flow prediction in an end-to-end way. By processing heterogeneous data in a shared 3D point flow space, PointPilot is able to absorb demonstrations from different embodiments. Extensive simulative and real world experiments validate our method. Notably, we demonstrate the zero-shot embodiment generalization in the PointPilot — one trained policy can be deployed onto previously unseen embodiments and successfully complete manipulation tasks, without any additional fine-tuning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Object-centric Prediction, 3D Point Flow Model, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4J8VOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PointPilot presents a coherent and integrated robotic manipulation pipeline for cross-embodiment policy learning. The work offers a well-motivated framework supported by a substantial set of simulation and real-world experiments. However, I am not yet convinced that the paper establishes a sufficiently significant methodological contribution for ICLR. My primary concerns are that the claimed novelties appear incremental compared to existing approaches, the experimental improvements are relatively marginal, and the ablation studies do not adequately substantiate the benefits of the proposed core components (see the weakness section for more details).

Critical strengths
  1. The paper presents an integrated pipeline for learning from heterogeneous action representations. Representing manipulation through task-relevant 3D motion rather than embodiment-specific actions is well motivated and offers a potentially useful approach to cross-embodiment learning.

  2. The cross-embodiment transfer results in Table 4 are encouraging, demonstrating the potential of the proposed representation across different robotic platforms and manipulation tasks.

  3. The paper and appendix provide sufficient implementation details to understand the proposed framework. I particularly appreciate the inclusion of real-world robotic experiments, which help demonstrate the practical feasibility of the approach. Overall, the paper presents a relatively comprehensive experimental study.

Critical weaknesses
  1. Presentation and clarity.
  • The methodology would benefit from more explicit connections between the text and the figures. For example, Figure 2 is not directly referenced or discussed in the main methodological descriptions, making it more difficult for readers to follow the overall pipeline.

  • The paper would benefit from a clearer distinction between conceptual novelty and system-level integration.

  1. Limited methodological novelty and insufficient justification through ablations.
  • The central idea of representing 3D object motion as point flow, including the robot gripper, has already been explored in related work. For example, PointWorld (https://arxiv.org/abs/2601.03782) also represents robotic embodiments and scene dynamics through 4D flow. Moreover, the fundamental distinction between PointPilot and existing approaches such as 3D Diffusion Policy remains unclear. I would appreciate a more explicit discussion of the methodological differences and the specific advantages introduced by PointPilot.

  • Regarding Table 1, I am not fully convinced that the selected features provide a fair comparison of methodological capabilities. For instance, Feature 2 (the absence of a separate grasping module) could potentially be incorporated into existing frameworks without fundamental architectural changes. Furthermore, the distinctions represented by Features 4 and 5 require clarification. Do these indicate that the existing methods were only trained or evaluated in single-arm or bimanual settings, or that they are fundamentally incapable of supporting the alternative setting, even after retraining? This distinction is important when assessing the claimed novelty.

  1. Marginal and inconsistent experimental improvements.
  • In Table 2, PointPilot exhibits a substantial performance drop on LIBERO-Goal compared with its performance on the other suites. The underlying reasons are insufficiently analysed. Moreover, the authors report a 96.0% success rate after excluding "three tasks", but the calculation and justification for this exclusion are unclear. Compared with SmolVLA, which demonstrates relatively consistent performance across the evaluated suites, the advantages of PointPilot are not convincingly established.

  • In Table 3, the improvement over existing methods is marginal, particularly compared with DP3 (56.6% versus 55.2%). Such a small performance difference provides limited evidence of the superiority of the proposed approach without further statistical analysis.

  1. Insufficient experimental validation and ablation studies.
  • In Table 4, X-VLA exhibits unusually poor performance on the "Click alarm clock" task with the ALOHA embodiment. I would appreciate an explanation of this observation. In fact, X-VLA performs reasonably well on the remaining seven tasks under ALOHA, which suggests that the results should be interpreted more carefully.

  • The cross-embodiment evaluation only includes X-VLA as a baseline. I believe this comparison is insufficient to support broader claims about superior cross-embodiment generalization. Additional representative baselines would strengthen the experimental conclusions.

  • More importantly, the experimental and ablation results do not clearly demonstrate the specific benefits of using a unified point-flow representation for both grippers and manipulated objects. At present, the proposed formulation appears primarily to provide an alternative representation for predicting robot and object motion. It remains unclear what intrinsic advantages this representation offers over existing approaches.

Questions for authors

Please see the weakness for the details of the questions that I would like to understand from the authors' explanations.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4J8V (Part I)

Thank you for your comments and suggestion which are really useful. Below is the corresponding reply to each weakness and question.


Reply to weakness 1:

Thank you for this suggestion. We referenced Fig. 2 in the method section and explained the pipeline through text and figures. We also clarified the distinction between the following contributions:

Conceptual contribution: PointPilot uses task-focused objects as its basic modeling units, and classifies each grounded object with an execution role. This design associates an object-centric model with different manipulation modes, and provides a unified motion representation across embodiments, supporting joint training on heterogeneous demonstrations with different embodiments and manipulation modes.

System integration: PointPilot integrates subtask planning, semantic grounding, object-flow prediction, and a geometric execution adapter. As the task progresses these components automatically select relevant objects, predict their motion, and convert it into robot actions. This system integration enables autonomous, closed-loop execution of multi-stage tasks. Compared with previous object-centric models that only predict the motion of a single fixed object, PointPilot automatically selects, switches and predicts multipled targeted objects, supporting closed-loop execution of tasks involving different objects within a unified framework.


Reply to weakness 2:

Methodological differences and additional experiments. Thank you for this detailed comment. We agree that representing robot and object motion in 3D has been explored previously, like pointworld. However, there still exists some differences.

Specifically, PointWorld predicts full-scene point flows conditioned on scene point clouds and robot actions. PointPilot instead only predicts subtask-relevant object flows from language and RGB observations, then converts them into executable actions. Besides, PointWorld requires stereo cameras to collect high quality depth images, which is time-consuming and time cost . Such stringent requirements impose high demands on the quality of pretraining data for PointWorld, rendering most existing datasets unusable and thereby limiting its scalability to large-scale pretraining. PointPilot constructs task-relevant object-flow supervision from RGB demonstration videos, together with camera calibration and recorded robot states, without requiring additional stereo-depth capture. This enables PointPilot to use robot demonstrations without depth observations, reducing dependence on depth-sensing equipment. As a VA policy, DP3 inputs point clouds and predicts robot actions, whereas PointPilot is able to accept language input and predicts 3D object flow. The distinction concerns the prediction target and execution mechanism, beyond the use of 3D information.

We additionally evaluated DP3 (EEF) for zero-shot cross-embodiment transfer. Both methods use ALOHA demonstrations for RoboTwin training and transfer to ARX and Piper with frozen weights; PointPilot retains its joint training with LIBERO. PointPilot achieves 75.6% and 71.0% success on ARX and Piper, respectively, compared with 23.9% and 10.1% for DP3. These results further support PointPilot’s transfer advantage on the evaluated tasks.

Comparison criteria in Table 1. We do think it is valuable to define our comparison criteria more precisely. Feature 2 describes whether a method covers pre-grasp and post-grasp manipulation through a unified motion interface without a separate grasping module. For previous object-centric frameworks which don't predict gripper states, incorporating grasping also requires representing gripper motion and handling phase transitions, potentially requiring additional prediction targets, supervision, and data processing beyond a simple model change.

Features 4 and 5 concern whether the original representation and execution interface directly support single-arm and bimanual manipulation, rather than whether a method could support another setting after extension. For example, an interface restricted to one arm cannot jointly trained on heteoregenous data. It requires nontrivial future work to change the model dimension and execution logic to support two arms. PointPilot unifies both modes through execution-role assignment and jointly trains one model on LIBERO and RoboTwin. Existing work didn't show this ability.

AuthorsOpenReview ↗

Response to Reviewer 4J8V (Part II)

Reply to weakness 3:

LIBERO-Goal performance. The lower success rate is concentrated in three tasks involving frictional contact without fixed gripper–object coupling. We believe this reflects their sensitivity to contact accuracy: errors in point-flow prediction and execution mapping can accumulate and, thus, prevent the gripper from maintaining contact, causing execution failures. The reported 96.0% is the unweighted average success rate over the remaining seven tasks, provided to analyze performance across contact modes.

Comparison with DP3. Although PointPilot’s RoboTwin success rate exceeds DP3’s by only 1.4 percentage points, their training settings differ. DP3 is a vision-action policy trained separately for each task without language input, whereas PointPilot jointly trains a single model on LIBERO and RoboTwin.

To further evaluate cross-embodiment capability, we added more zero-shot transfer experiments including DP3 (EEF). Both methods use ALOHA demonstrations for RoboTwin training and subsequently transfer to ARX and Piper with frozen weights. PointPilot achieves 75.6% and 71.0% success on ARX and Piper, respectively, compared with 23.9% and 10.1% for DP3. These results further support PointPilot’s zero-shot cross-embodiment transfer advantage beyond DP3.

AuthorsOpenReview ↗

Response to Reviewer 4J8V (Part Ⅲ)

Reply to weakness 4:

Additional baselines. Beyond X-VLA, we added π₀.₅ (joint), π₀.₅ (EEF), and DP3 (EEF), covering both joint-angle and end-effector-pose action representations. These experiments follow the protocol of source training on ALOHA followed by zero-shot evaluation on ARX and Piper with frozen weights. Below are the detailed experiment results:

π₀.₅ (Joint) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA100100100988786797490.50–
ARX9817984542729.38−61.13
Piper000–00000.00−89.43

π₀.₅ (EEF) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA98100100749285866787.75–
ARX1677350304233134.88−52.88
Piper648071–840351049.14−40.57

DP3 (EEF) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA100981008710083636386.75–
ARX87473186001223.88−62.88
Piper1843–10003610.14−76.57

PointPilot (Point Flow) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA98100947579100548886.00–
ARX859390697567418575.63−10.38
Piper797583–7769397571.00−16.57

Results Revision and task retraining. Thank you for pointing out this discrepancy. With the released official X-VLA checkpoint, we obtained a 17% success rate over 100 ALOHA trials on click_alarmclock. In the failed scene, we observed that the gripper would approach the alarm clock correctly, but couldn't approach deep enough to press the button------the distance between the final gripper and the clock is from 5mm to 10mm. Thus, we think there might be some problem with the official released checkpoint. Then, we retrain X-VLA on this task with our own rolled-out data and get 96% success rate which coincides with the officially reported success rate. And other tasks' success rates also match reported one. With this new checkpoint, the cross embodiment success rate on ARX reduces from 62% to 60%. The table result has been updated.

Contribution of the unified point-flow representation. First, task-relevant object motion serves as a shared prediction target. This decouples task-motion prediction from embodiment-specific execution without requiring a separately learned action decoder for each embodiment. Experiments demonstrate the tasked-focused object motion a better cross-embodiment ability than other action representation, such as end-effector pose or joint action space. Furthermore, including grippers as task-focused objects allows the same interface to cover both pre-grasp gripper motion and post-grasp object motion. Combined with execution-role assignment and subtask planning, this enables automatic target switching, closed-loop and long horizon/multiple stages manipulation in both single-arm and bimanual settings. To our best knowledge, existing object-centric methods (mentioned in the related work) possess limited ability to achieve this goal.

I sincerely thank the authors for their detailed responses and additional experiments, which address several of my concerns regarding the generalization and robustness of the proposed method. In particular, the expanded cross-embodiment evaluations provide stronger evidence of its practical effectiveness. However, my primary concern regarding methodological novelty remains. Compared with existing approaches, PointPilot appears to introduce relatively incremental modifications, mainly through extending the input representations and integrating additional modalities and existing components, rather than proposing a new methodology/framework. While I acknowledge the value of this system-level integration, I remain unconvinced that the paper demonstrates sufficiently substantial methodological contributions for ICLR, particularly given the lack of controlled ablations isolating the benefits of its core unified point-flow representation. Therefore, despite appreciating the authors' efforts and the improved experimental evidence, I have decided to maintain my original rating.

Reviewer_h7nnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PointPilot, which uses the 3D point flow of task-relevant objects as a shared motion prediction interface across different robots. Depending on the manipulation phase, the method selects either the gripper itself or an object coupled to it as the prediction target, and assigns execution roles to the left and right hands. It combines subtask planning with textual memory, 3D object localization, and multi-view geometric alignment to predict future point displacements and gripper opening and closing. Under the assumption of rigid coupling within each phase, a non-learned geometric adapter converts point flows into end-effector poses, which are then executed using the robot's kinematics and controller. The authors validate the framework through LIBERO, RoboTwin, zero-shot cross-robot transfer in simulation, and three real-world robot tasks.

Critical strengths
  1. Transforming different robots' native action spaces into a shared object motion space helps handle heterogeneous demonstrations. Treating the gripper as an object as well, and switching prediction targets with the manipulation phase, addresses the difficulty of covering pre-grasp actions when predicting only the manipulated object. The execution roles further connect single-arm and dual-arm manipulation, while rigid-body fitting avoids the need to additionally learn a continuous end-effector action decoder. Together, these designs constitute a practically meaningful contribution.
  2. The experiments provide relatively comprehensive coverage and valuable evidence of cross-robot transfer. The same policy achieves 85.8% on LIBERO and 56.6% on RoboTwin, supporting the feasibility of joint learning for single-arm and dual-arm manipulation. On Piper, PointPilot achieves 71.0%, a decrease of 16.6 percentage points relative to the source configuration on the same seven tasks, whereas X-VLA drops by 64.7 percentage points, indicating a relatively clear transfer advantage.
Critical weaknesses
  1. The experiments do not yet sufficiently isolate the contribution of the object point flow representation itself. The existing ablations mainly evaluate planning memory and the effects of 3D encoding and multi-view alignment on the point flow learning loss. A controlled comparison that preserves the same planning, geometric inputs, and training configuration while replacing only the motion output representation is missing.
  2. The evidence for generalization and unified manipulation still has clear boundaries of applicability. The cross-robot evaluation is restricted to paired simulation scenarios that are feasible for the expert, with additional alignment of Piper's initial end-effector pose. The current results support a unified interface and transfer under these controlled conditions, but are not yet sufficient to support transfer capabilities across a broader range of contact modes or under natural initial conditions.
  3. In line 155, cross-view fusion ,: remove the space before the comma.
Questions for authors
  1. Is there a controlled comparison that retains the same VLM planning, multi-view geometric conditioning, training data, and budget, and changes only to direct prediction of end-effector poses? If so, please report its task success rate and cross-robot transfer results to help assess the contribution of the proposed object motion interface.
  2. Please report the number of candidate scenarios in the cross-robot experiments, the acceptance rate at each filtering step, and how many Piper scenarios were excluded because initial end-effector alignment could not be completed. If results on sensitivity to initial pose deviations are already available, please also describe them to help assess the range of initial conditions covered by the transfer results.
  3. The authors primarily attribute the lower performance to three uncoupled, friction-dependent tasks. However, they have not sufficiently analyzed whether failures on these tasks mainly occur in motion prediction or execution mapping. Could the authors provide an analysis of representative failures to clarify whether the main bottleneck lies in motion prediction or execution mapping?
1 comment
AuthorsOpenReview ↗

Response to Reviewer h7nn

Thank you for your comments and suggestion which are really useful. Below is the corresponding reply to each weakness and question.


Reply to weakness 1 and question 1:

Thank you for mentioning the contribution of the motion output representation, which is helpful to demonstrate the significance and uniqueness of PointPilot. Since PointPilot is a 3D object-centric model and point-flow prediction is a deliberate core design, direct change output to EEF-pose prediction for ablation may change this core motion interface. To further compare the object point flow representation with the EEF and the joint action space, we added extra three baselines (which predict EEF-pose/joint) beyond X-VLA: π₀.₅ (joint), π₀.₅ (EEF), and DP3 (EEF). PointPilot achieves stronger zero-shot cross-embodiment transfer than these baselines on the evaluated tasks.

π₀.₅ (Joint) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA100100100988786797490.50–
ARX9817984542729.38−61.13
Piper000–00000.00−89.43

π₀.₅ (EEF) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA98100100749285866787.75–
ARX1677350304233134.88−52.88
Piper648071–840351049.14−40.57

DP3 (EEF) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA100981008710083636386.75–
ARX87473186001223.88−62.88
Piper1843–10003610.14−76.57

PointPilot (Point Flow) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA98100947579100548886.00–
ARX859390697567418575.63−10.38
Piper797583–7769397571.00−16.57

The detailed results have been added in Table 4.


Reply to weakness 2 and question 2:

PointPilot’s prediction network primarily relies on visual grounding to localize grippers. In our RoboTwin setup, Piper’s grippers are initially outside the main-camera view. We therefore align its initial end-effector poses to make grippers visible in the first frame and satisfy the model’s visual observation requirements. So PointPilot doesn't necessarily require the same initial gripper pose.

To support this claim, we made another cross-embodiment experiment with different aloha and piper initial gripper poses but still both visible in the first frame. With different initial gripper poses, PointPilot's cross-embodiment success rates reduce from 86% to 68.9, whereas with the same gripper pose, the success rates are from 86% to 71%. Thus, different initial poses won't affect cross-embodiment ability. The observed difference is within the expected run-to-run variability.

In RoboTwin’s official task suite, task availability differs across embodiments, with some tasks unavailable for ARX or Piper. We therefore evaluate only tasks supported by both the source and target robots. Before policy evaluation, we retain scenes in which official expert policies succeed on both embodiments.


Reply to weakness 3:

Thank you for pointing this out. We have removed the extra space before the comma in “cross-view fusion ,” .


Reply to question 3:

Thank you for noticing these friction-dependent tasks. Here is some detailed analysis. These friction-dependent tasks without fixed gripper–object coupling are somehow sensitive to contact accuracy. Contact accuracy will decrease due to both the point-flow prediction and the adapter conversion. Errors in motion prediction and point-flow-to-EEF mapping can compound, and both contribute to the failure of these tasks.

For example, moving a plate with a two-finger gripper requires sustained contact to transmit force effectively. Even a few millimeters of error in the resulting end-effector position can weaken or break contact, preventing the plate from moving as intended and causing task failure. The detailed explanantion has been discussed in 5.1 (simulation results).

Reviewer_oCGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PointPilot predicts the motion of a small set of task-focused 3D points, including points on the grippers, and turns the predicted rigid motion into end-effector commands with a geometric adapter. A VLM handles subtask planning, while 3D grounding assigns an object and execution role to each arm. The ambition is appealing: the same action representation should work for single-arm and bimanual tasks, and even transfer to a robot that was not represented in policy training.

Critical strengths
  1. The ALOHA-to-Piper test is the result that made me take the paper seriously. On seven compatible tasks, PointPilot has a much smaller transfer gap than the reported X-VLA checkpoint (Table 4). The policy weights stay fixed, and the authors use paired expert-feasible scenes. Even with the selection caveat below, this is a more direct test of embodiment transfer than simply reporting performance on two robots trained separately.

  2. The object-point-flow interface is a useful design choice. It separates predicted motion of task-relevant objects from robot-specific conversion into end-effector commands, instead of asking a single action head to absorb both. Treating grippers as points in the same representation is a coherent way to cover single-arm and bimanual tasks. This design gives a plausible reason the model could transfer, even though the experiment below does not yet isolate that reason.

  3. The planning-memory ablation is concrete: LIBERO-Long success rises from 55.0% to 77.2% (Table 5), addressing the tendency to repeat a completed subtask.

Critical weaknesses
  1. The scope of the transfer evidence is narrower than the “unified” framing suggests. Table 4 covers eight selected simulation tasks, only seven compatible with Piper, under expert-feasible paired initializations (Appendix C.3). There is no physical cross-robot transfer test. This is a useful controlled setting, but I cannot tell how much of the result survives a broader task sample or less curated starting states.

  2. The paper does not isolate the cause of the transfer result. Spatial alignment is mainly ablated using point-flow loss (Fig. 4), not target-robot success, and the effects of grounding and the rigid adapter are not separated on the ARX/Piper tasks. A lower prediction loss does not tell me which component prevents failures after a robot change. This missing success-rate ablation bears directly on the main mechanism.

  3. The comparative and physical evidence needs more calibration. Source-benchmark baselines mostly come from published numbers rather than matched retraining; on RoboTwin Clean, 56.6% is only 1.4 points above the reported DP3 result (Table 3), without run-to-run variation. The physical tasks have no comparative baseline or uncertainty estimate, and QR scanning succeeds in 10/20 trials (Fig. 3). These results show feasibility, but not a robust advantage over alternative systems.

  4. The rigid-coupling adapter may be brittle in pouring or regrasping, but the physical section gives no failure breakdown to show how often this assumption matters.

Questions for authors
  1. On the same ARX/Piper scenes as Table 4, what are the target-robot success rates without multi-view alignment and without the object-flow adapter? If those runs are unavailable, please narrow the claim about which component enables transfer.
  2. How many scenes were rejected by the expert-feasibility and pose-alignment filters, by task and robot? Also, is the RoboTwin result stable across training seeds?
5 comments
AuthorsOpenReview ↗

Response to Reviewer oCGT (Part I)

Thank you for your comments and suggestion which are really useful. Below is the corresponding reply to each weakness and question.


Reply to weakness 1 and question 2:

In our RoboTwin setup, we align the first pose of each embodiment only because piper’s grippers are initially outside the main-camera view. Since PointPilot’s prediction network relies on visual grounding, we align Piper’s initial end-effector poses to make the grippers visible in the first frame.

In RoboTwin’s official task suite, task availability differs across embodiments, with some tasks unavailable for ARX or Piper. We therefore evaluate tasks supported by both the source and target robots, and use expert policies to retain scenes that both robots can successfully complete. The expert feasibility and task success filters all follow the standard code of the official setting and other mainstream methods.

Thank you for your question regarding the stability of our method across different training seeds. To further investigate this issue, we evaluated our method on RoboTwin using three independent training seeds under identical training and evaluation settings. Our originally reported success rate is 56.6%. For reference, the results are as below:

Seed 0 (Original)Seed 1Seed 2
56.658.257.9

Mean ± Std: 57.57 ± 0.85


Reply to weakness 2 and question 1:

Besides the lower point loss shown in figure 4, we added furthermore ablation study on spatial alignment. On LIBERO and Robotwin, incorporating 3D spatial alignment improves the success rate by 3% and 5.4% separately. These results support its effectiveness in improving motion prediction and task execution and the results have been updated in table 6.

In terms of causes of the transfer result, we added more textual explanation in 5.2 (cross-embodiment part). Notably, PointPilot doesn't predict or train the EEF/joint action space and, instead, uses task-focused object motion as a shared prediction target. This reduces dependence on the embodiment itself. The reason why the robot can complete tasks based on point flow is that the rigid adapter can convert the task-focused object (including gripper) motion to EEF. Hence, the grounding module and the rigid adapter cannot be excluded from PointPilot, otherwise it will fail in all experiments. However, in order to further demonstrate the advantage of task-focused object motion and the difference between object motion and EEF/joint, we evaluated more methods including pi0.5 (joint), pi0.5 (EEF) and DP3 (EEF) in cross-embodiment experiments.

Below are the results:

π₀.₅ (Joint) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA100100100988786797490.50–
ARX9817984542729.38−61.13
Piper000–00000.00−89.43

π₀.₅ (EEF) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA98100100749285866787.75–
ARX1677350304233134.88−52.88
Piper648071–840351049.14−40.57

DP3 (EEF) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA100981008710083636386.75–
ARX87473186001223.88−62.88
Piper1843–10003610.14−76.57

PointPilot (Point Flow) :

RobotGrab rollerShake bottleShake bottle horiz.Click alarm clockClick bellDump bin into big binStack two bowlsPick dual bottlesAverageΔ (pp)
ALOHA98100947579100548886.00–
ARX859390697567418575.63−10.38
Piper797583–7769397571.00−16.57

Overall, PointPilot achieves stronger zero-shot cross-embodiment transfer than these baselines on the evaluated tasks, demonstrating the superiority of task-focused object point flow. The results have been updated in Table 4.

AuthorsOpenReview ↗

Response to Reviewer oCGT (Part II)

Reply to weakness 3:

Although PointPilot’s success rate on RoboTwin is only 1.4 percentage points higher than DP3’s, their training settings differ. DP3 is a vision-action (VA) policy without language conditioning and uses a separately trained model for each task. In contrast, PointPilot, as a 3D object-centric model, is jointly trained on LIBERO and RoboTwin, using a single policy for both single-arm and bimanual tasks.

To further assess cross-embodiment generalization, we additionally evaluated DP3 on the transfer tasks. PointPilot achieves success rates of 75.6% on ARX and 71.0% on Piper, compared with 23.9% and 10.1% for DP3, respectively. These results provide further evidence of PointPilot’s transfer advantage than DP3.


Reply to weakness 4:

Our adapter assumes stage-local rigid coupling. During pouring, this assumption concerns the relative pose between the gripper and the container; tilting a securely grasped container does not itself violate the assumption. In the real experiment, we also use PointPilot to pour water, achieving 70% success rate. Regrasping requires a transition between coupling relationships: gripper flow guides approaching and grasping, while object motion guides execution after a stable grasp.

Thank you for the additional experiments. The three-seed results address my question about PointPilot’s training stability, and the new baselines strengthen the evidence for system-level cross-embodiment transfer. However, I still have concerns about the attribution and scope of this result. The reported alignment ablations do not explicitly provide target-robot success rates on the same ARX/Piper scenes as Table 4, and comparisons between complete systems do not isolate which components enable transfer. I appreciate that simply removing the adapter would make the system nonfunctional; without a controlled alternative, the component-level claims should be narrowed. Additionally, the response explains the rationale for scene selection but does not report the requested candidate and rejected scene counts by task and robot. This leaves the extent of selection bias unclear. Overall, the new evidence strengthens the empirical result, but the mechanism behind the transfer advantage and its generality beyond the selected feasible scenes remain insufficiently established.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Dear Reviewers,

We sincerely thank the reviewers for their thoughtful and constructive feedback. Their comments have helped us strengthen the experimental evaluation and clarify the contributions and limitations of PointPilot. We summarize the main revisions below. The principal additions and revised explanations are marked in blue in the revised manuscript.

  1. Expanded zero-shot cross-embodiment evaluation. We have added three baselines—π₀.₅ (joint), π₀.₅ (EEF), and DP3 (EEF)—to the cross-embodiment evaluation in Section 5.2 and Table 4. These comparisons cover both joint-space and end-effector-space action representations under transfer from ALOHA to ARX and Piper. PointPilot achieves average success rates of 75.63% on ARX and 71.00% on Piper, compared with 23.88% and 10.14% for DP3, respectively. The expanded evaluation provides additional evidence of PointPilot’s transfer capability across the evaluated embodiments and tasks.

  2. Additional success-rate ablation of spatial alignment. To complement the point-flow loss analysis, we have added an ablation measuring the effect of spatial alignment on task success in Section 5.4, Table 6. Incorporating spatial alignment improves success from 82.8% to 85.8% on LIBERO and from 51.2% to 56.6% on RoboTwin, corresponding to gains of 3.0 and 5.4 percentage points. These results directly connect the geometric design to improvements in task execution.

  3. Clearer methodological comparisons and explanation of the framework. We have expanded the discussion in Section 2.1 to explain the requirements of supporting multiple task-focused objects and assigning their motion to different grippers in bimanual manipulation. We have also clarified the criteria underlying Table 1 in our responses, including the integration of grasping and the demonstrated support for single-arm and bimanual execution. These revisions make the distinction between PointPilot an other object-centric methods clearer.

  4. More detailed analysis of experimental results. We have expanded Section 5.1 to analyze the three friction-driven tasks that limit LIBERO-Goal performance, explaining how errors in point-flow prediction and geometric conversion can compound and disrupt sustained contact. In Section 5.2, we further explain how gripper flow guides pre-grasp motion and how grasped-object flow supports transfer across embodiments with different end-effector trajectories. We have also added the Policy setup column to Table 3, making the task-specific training of baselines such as DP3 and PointPilot’s shared-policy setting explicit.

We appreciate all reviewers’ careful assessment and the opportunity to strengthen our work. We hope these additional experiments and clarifications address the main concerns. Look forward to more reply and comments to strengthen this work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.