acceptodds
Under review as a conference paper at ICLR 2027

Unitas: A 3D-Native World Action Model for Embodied Manipulation

10 views

Abstract

World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world? Most existing WAMs build on pretrained video generators and represent world evolution through images or visual latents. Robotic interaction, however, takes place in metric three-dimensional space, while images are view-dependent projections whose pixel distances do not directly encode physical distances. We introduce **Unitas**, to our knowledge the first 3D-native world action model that unifies observations, actions, and scene dynamics in a shared metric 3D frame within each interaction, using a common representation across robot embodiments and human hands. Action flow represents human hands and robot grippers as 3D point trajectories, while scene flow describes scene-point displacements conditioned on these trajectories. World-aligned 3D positional embeddings ground visual tokens with or without depth input, and a physical-time trajectory tokenizer encodes each point trajectory as one token anchored at its current 3D position. This interface supports both direct action execution and action-conditioned scene prediction. With 1.7B parameters, Unitas achieves the best action-conditioned scene prediction on RoboTwin among the compared methods, with up to 49% lower displacement errors than PointWorld, and state-of-the-art manipulation success, including 99.8% on LIBERO and an average of 85% across real-world tasks. Code will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D world models, world action models, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_J9SaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces UNITAS, a 3D-native world action model for embodied manipulation that represents visual observations, robot and human-hand motion, and scene dynamics within a shared metric coordinate frame. It represents agent motion as action flow and environmental changes as action-conditioned scene flow. World-aligned positional embeddings ground visual features with or without observed depth, while a physical-time trajectory tokenizer encodes each point trajectory into a spatially anchored token. Coupled diffusion transformers predict point flows and executable robot commands, supporting both action-conditioned scene prediction and policy-only inference without future-flow generation. The 1.7B-parameter model is evaluated on LIBERO, LIBERO-Plus, RoboTwin, VLABench, and two real-robot tasks. Reported results include 99.8% success on LIBERO, 89.1% on LIBERO-Plus, and approximately 35% lower moving-point trajectory error than PointWorld on RoboTwin.

Critical strengths
  1. The paper presents a coherent 3D representation that places visual observations, robot and human-hand motion, and scene changes in a shared metric coordinate frame within each interaction window, rather than treating depth merely as an additional input. Action flow, scene flow, and spatially anchored trajectory tokens form a unified framework grounded in manipulation geometry. This is a valuable conceptual contribution.

  2. The evaluation is broad, covering single-arm and bimanual manipulation, robustness to distribution shifts, real-robot execution, and action-conditioned scene prediction. This breadth provides evidence for the framework’s effectiveness across multiple settings.

  3. The model demonstrates strong quantitative results in 3D scene prediction. Compared with PointWorld, which also directly predicts 3D trajectories, it reduces Moving ADE from 0.656 cm to 0.427 cm, an approximately 35% reduction, while improving Moving Acc@1cm from 88.22% to 92.31%.

  4. The attention structure and key–value caching enable efficient policy-only inference, allowing the model to generate executable commands without predicting future point flows. The reported latency of 55.77 ms per action chunk on an RTX 5090 D highlights the practical value of this design.

Critical weaknesses
  1. Line 106 reports a success rate of 65% for Pack the Ball, whereas Table 2 and Section 4.3 report 80% (16/20).

  2. According to Eq. (3) and Section 3.5, action flow (A) and executable commands (u) are generated separately given a shared context, using independent noise, while scene prediction (S) is conditioned on (A) as its sole action input. This design does not explicitly ensure consistency between the generated action flow and the motion resulting from executing the generated commands, particularly when multiple valid actions exist for the same task. Appendix F.2 provides indirect evidence by comparing predicted and observed scene changes, but does not directly quantify the discrepancy between the two action representations. I suggest comparing the generated action flow with the actual gripper-point trajectories obtained by executing the generated commands, and reporting both trajectory and endpoint errors. This would more directly assess whether the predicted scene consequences correspond to the actions actually executed, strengthening the case for unified world–action modeling.

  3. Table 3 shows that removing action-flow supervision or history tokens reduces overall success by 6.85 and 9.82 percentage points, respectively, whereas adding scene-flow supervision to the reference model containing both components yields only a 0.73% improvement, from 85.74% to 86.47%. These ablations therefore provide stronger evidence for the value of geometric action supervision and motion history in policy learning than for the additional benefit of scene-dynamics supervision. Since scene flow is not generated during policy-only inference, I encourage the authors to distinguish clearly between the value of scene prediction as an auxiliary training objective and the necessity of generating scene predictions at inference time.

Questions for authors

Please refer to the "Critical Weaknesses" for specific issues. If these can be resolved, I will raise the score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer J9Sa (Part I)

We appreciate your recognition of our unified 3D representation, broad evaluation, and efficient policy-only inference. Your suggestions helped us clarify the role of scene prediction and provide direct evidence of action–execution consistency.

W1. Pack the Ball success rate

Thank you for catching this typo. We have corrected the introduction to 80% (16/20), consistent with Table 2 and Section 4.3.

W2. Direct agreement between action flow and executed motion

We appreciate the reviewer’s suggestion to directly evaluate action–execution consistency. Although the loss does not explicitly enforce kinematic agreement, the two branches share context and are trained against corresponding demonstration targets. We added the requested execution-based comparison.

We execute the generated commands and reconstruct gripper trajectories from the actual joint states using URDF forward kinematics. We compare these with predicted action flow in the same coordinate frame, matching point identities and executed steps. Correlations use displacements relative to their common initial positions.

BenchmarkCCC ↑Pearson ↑ADE ↓ (mm)FDE ↓ (mm)
RoboTwin, 50 tasks0.94660.95119.4116.53
LIBERO0.98320.98624.926.84

ADE measures mean trajectory error, and FDE measures error at the last executed step of each prediction window. Pearson measures displacement correlation, while CCC additionally accounts for differences in mean and scale. Across both benchmarks, Pearson correlations exceed 0.95 and ADE remains below 1 cm, indicating close agreement between predicted action flow and executed motion under the evaluated protocol.

W3. Training-time and inference-time contributions of scene prediction

Thank you for highlighting this distinction. Our original policy-only results evaluate the training-time contribution of scene supervision; future action flow and scene flow are not generated during policy-only inference. We now provide additional supervision ablations and evaluate an optional scene-guided action-selection procedure.

1. Training-time scene supervision

To accelerate training and reduce GPU memory usage, our default setting drops 99% of scene points after voxelization and capping, retaining only 1% for supervision. This sparse supervision may partly explain the modest LIBERO-Plus gain from 85.74% to 86.47% (Section 4.4, Table 3, L417–431). Fast-WAM [1] similarly finds that future-video co-training benefits action learning.

We added a RoboTwin ablation comparing no scene supervision with drop rates of 0.99, 0.90, and 0.75, retaining 1%, 10%, and 25% of scene points, respectively. All configurations are evaluated at their 60k-iteration checkpoints. The main RoboTwin results in Table 1 use 200k-iteration checkpoints; extending all ablation runs to 200k iterations was not feasible within the available computational budget and rebuttal period. Trajectory and position errors are in mm, rotation errors in degrees, and success rates in percent.

ConfigurationAction-flow mean ↓Action-flow final ↓EEF position mean ↓EEF position final ↓EEF rotation mean ↓Closed-loop success ↑
No scene supervision8.8512.734.818.301.8274.6
Drop rate 0.995.308.274.267.281.4278.3
Drop rate 0.905.869.054.357.241.3979.2
Drop rate 0.755.549.484.437.401.4178.9

Scene supervision reduces all reported prediction errors and improves closed-loop success by 3.7–4.6 percentage points across the tested drop rates. Retaining 10% or 25% of scene points provides additional success gains of 0.9 and 0.6 percentage points over the default 1% setting.

AuthorsOpenReview ↗

Response to Reviewer J9Sa (Part II)

2. Inference-time action selection

To evaluate the additional value of consulting predicted scene dynamics, we independently sample K=4K=4 command chunks and predict scene flow under each candidate’s associated action-flow condition. We fit rigid object poses to the predicted points and render the available simulator meshes at those poses. Qwen3-VL-4B-Instruct [2] evaluates these rendered outcomes against the current observation and instruction to select a command chunk. After execution, the procedure repeats with the updated observation.

With 100 trials per task and method, the full scene-guided selection procedure improves success on all three tasks, increasing mean success from 71.67% to 79.67% (+8 percentage points).

RoboTwin taskDirect policy (%)Scene-guided best-of-4 (%)Gain (pp)
hanging_mug6174+13
put_object_cabinet7782+5
place_can_basket7783+6
Mean over all 3 tasks71.6779.67+8.00

These results demonstrate an additional use of generated scene dynamics for action selection, while scene generation remains optional for policy execution.

Manuscript updates

The revised manuscript includes the following additions.

  • W1 — Corrected success rate in Section 1.
  • W2 — Direct action–execution agreement in Appendix H.1, Table 15.
  • W3 — Point subsampling in Section 3.4 and supervision ablation in Appendix H.2, Table 16. Action selection in Section 4.5, Figure 6 and Table 5, and Appendix H.3.

References

[1] T. Yuan et al. Fast-WAM: Do World Action Models Need Test-Time Future Imagination? 2026, Section 4.3.2. arXiv:2603.16666

[2] S. Bai et al. Qwen3-VL Technical Report. 2025.

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer J9Sa,

Based on your valuable feedback, we have supplemented our experiments to analyze the consistency between action flow and action command, and the role of scene flow prediction during training-time and test-time stages. We have also updated the revision as you suggested. As we approach the end of the discussion period, we would be grateful if you could review our response and consider our revised manuscript. Your constructive comments have helped us to significantly improve our work, and we believe that we have thoroughly addressed your concerns.

Thank you very much for your time and detailed review. We welcome any further additional questions or requests for clarification.

Yours sincerely,

The Authors

Reviewer_LuaCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents UNITAS, a 3D-native world action model that unifies robot actions and scene dynamics within a shared metric 3D space. The proposed framework represents robot grippers, human hands, and scene dynamics through 3D point trajectories, enabling joint action generation and action-conditioned scene prediction. To support this formulation, the authors introduce world-aligned 3D positional embeddings and a physical-time trajectory tokenizer, together with a Mixture-of-Transformers architecture for modeling point flows and executable robot commands.

The method is evaluated on several simulation benchmarks, including LIBERO, RoboTwin 2.0, VLABench, and LIBERO-Plus, as well as two real-world manipulation tasks. The results demonstrate strong performance in both policy execution and action-conditioned scene prediction.

Critical strengths

The paper presents an interesting and well-motivated approach to world action modeling. Representing both robot motion and scene dynamics as 3D point trajectories provides a physically meaningful alternative to conventional video-based world models. The proposed world-aligned positional embeddings and physical-time trajectory tokenizer are also useful designs for handling heterogeneous observations and motion data across different embodiments.

The experimental evaluation is relatively comprehensive, covering multiple simulation benchmarks, real-world manipulation tasks, and action-conditioned scene prediction. UNITAS achieves competitive results against recent VLA and WAM methods, particularly on LIBERO, RoboTwin 2.0, and LIBERO-Plus. The ablation studies also provide evidence for the benefits of 3D spatial grounding, action-flow supervision, and motion history.

Critical weaknesses

1. Insufficient analysis of the consistency between action representations and the contribution of human demonstrations.

While UNITAS jointly models action flow and executable robot commands, the two outputs are generated through separate branches, and the attention mask prevents the action expert from directly accessing future flow tokens. This raises concerns about whether the predicted action flow accurately corresponds to the motion produced by the executable commands. Although Appendix F.2 provides an action–scene consistency evaluation, a direct analysis of the consistency between these two action representations is still missing. Additionally, the paper emphasizes a unified representation for human hands and robot grippers, but does not isolate the contribution of human demonstrations during pretraining. It remains unclear whether incorporating human-hand data provides measurable benefits for robot policy learning.

2. Limited evaluation of real-world generalization and robustness.

Although UNITAS demonstrates strong performance on two real-world manipulation tasks, the evaluation is limited to 20 trials per method per task, without a clear analysis of out-of-distribution robustness. While the paper reports promising robustness results on LIBERO-Plus, it remains unclear whether these improvements translate to real-world scenarios involving changes in object configurations, scene layouts, or camera viewpoints. More diverse real-world evaluations and failure-case analysis would strengthen the evidence for the method's generalization capabilities.

Questions for authors

Q1. Consistency between action flow and executable robot commands.

The action flow and executable commands are generated by separate experts, while the attention mask prevents the action expert from directly accessing future flow tokens. Although Appendix F.2 evaluates action–scene consistency, could the authors provide a more direct evaluation of the consistency between predicted action flow and the actual gripper trajectories produced by executable commands? It would also be interesting to investigate whether allowing the action expert to attend to predicted action-flow features could improve action consistency or policy performance, and how this would affect inference efficiency.

Q2. Contribution of human demonstrations.

The paper emphasizes the unified representation of human hands and robot grippers, but the current experiments only demonstrate the overall benefits of mixed-dataset pretraining. In particular, gripper trajectories are constructed from points sampled on the gripper geometry, whereas human-hand trajectories are derived from MANO keypoints, resulting in substantially different spatial distributions and motion characteristics. It therefore remains unclear how effectively the model learns transferable knowledge from these heterogeneous data sources, and to what extent human demonstrations actually contribute to downstream robot policy performance. Could the authors provide an ablation comparing robot-only pretraining with robot-and-human pretraining, ideally under comparable training budgets? This would help clarify whether human demonstrations provide additional benefits for robot policy learning beyond simply increasing the diversity of the training data.

Q3. Real-world OOD robustness and failure analysis.

The real-world results are promising, but it is unclear whether the evaluations include out-of-distribution (OOD) scenarios. Could the authors clarify whether the reported results cover variations in object positions, scene layouts, or camera viewpoints beyond the training distribution? If not, additional OOD evaluations would help demonstrate whether UNITAS maintains its advantages over π0.5\pi_{0.5} and Fast-WAM in more challenging real-world settings. Representative failure cases would also help explain the differences in robustness among these methods.

Q4. Applicability to dexterous manipulation.

The proposed 3D point trajectory representation is designed to accommodate different embodiments, including human hands. However, the current robotic evaluations mainly focus on single-arm and bimanual gripper manipulation. How would the proposed action-flow representation and action expert extend to dexterous hands with higher-dimensional action spaces and more complex contact interactions? An evaluation on a dexterous manipulation benchmark would help demonstrate the applicability of UNITAS to more complex robot embodiments.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LuaC (Part I)

We appreciate your careful review and constructive suggestions. We have supplemented additional experiments and clarifications to address your concerns.

Q1 & W1. Action-flow consistency

We appreciate the reviewer’s suggestion to directly evaluate action–execution consistency. Although the loss does not explicitly enforce kinematic agreement, the two branches share context and are trained against corresponding demonstration targets.

We execute the generated commands and reconstruct gripper trajectories from the actual joint states using URDF forward kinematics. We compare these with predicted action flow in the same coordinate frame, matching point identities and executed steps. Correlations use displacements relative to their common initial positions.

BenchmarkCCC ↑Pearson ↑ADE ↓ (mm)FDE ↓ (mm)
RoboTwin, 50 tasks0.94660.95119.4116.53
LIBERO0.98320.98624.926.84

ADE measures mean trajectory error, and FDE measures error at the last executed step of each prediction window. Pearson measures displacement correlation, while CCC additionally accounts for differences in mean and scale. Across both benchmarks, Pearson correlations exceed 0.95 and ADE remains below 1 cm, indicating close agreement between predicted action flow and executed motion. Allowing the action expert to attend to predicted action-flow features could strengthen this coupling, but would introduce an additional inference dependency. Its effects on consistency, closed-loop success, and end-to-end latency require joint evaluation; this architectural extension remains future work.

Allowing the action expert to attend to predicted action-flow features might strengthen this coupling, but would introduce an additional inference dependency. Its effects on consistency, closed-loop success, and end-to-end latency require joint evaluation, and this architectural extension remains future work.

Q2 & W1. Contribution of human demonstrations

Thank you for suggesting this controlled comparison. We compare robot-only and robot-and-human pretraining under matched pretraining and fine-tuning budgets.

Pretraining dataRoboTwin Demo-Clean success (%)RoboTwin Demo-Randomized success (%)
Robot only92.3391.98
Robot + human92.9493.04
Gain from adding human data (pp)+0.61+1.06

Human hand landmarks and gripper samples retain their different geometries, while shared metric displacements and physical-time coordinates provide a common motion interface. Adding human demonstrations improves RoboTwin success by 0.61 percentage points on Demo-Clean and 1.06 points on Demo-Randomized. This comparison measures the overall contribution of the human-data mixture under matched budgets, including the additional diversity it introduces.

Q3 & W2. Real-world OOD robustness and failure analysis

We appreciate the reviewer’s suggestions on real-world OOD evaluation and failure analysis. Our hardware tests cover OOD variations in object positions, with 20 trials per method on each of two tasks. The head-mounted camera configuration remains largely fixed relative to the manipulation workspace; these trials do not systematically evaluate camera or layout perturbations.

On LIBERO-Plus (Appendix C, Table 11), UNITAS achieves 89.1% overall success across seven perturbation types, compared with 85.7% for π0.5\pi_{0.5} [1] and 51.5% for Fast-WAM [2]. It also outperforms both methods under camera and layout perturbations. These results provide evidence of robustness in simulation. We agree that broader hardware OOD evaluations and systematic per-method failure analysis would strengthen the evidence. These evaluations, including representative failure cases, could not be completed within the limited rebuttal period and remain important directions for future work.

AuthorsOpenReview ↗

Response to Reviewer LuaC (Part II)

Q4. Extension to dexterous manipulation

Thank you for highlighting dexterous manipulation as an important extension. Our core contribution is a unified 3d world action model formulation: motions from different robot configurations, dexterous hands, and human hands can be represented as 3D point trajectories, while scene dynamics uses the same representation, aligning action and scene motion in the backbone’s input–output space. Our existing experiments support the utility of this shared formulation for robot control, including the matched-budget human-data ablation showing gains from incorporating human demonstrations.

Dexterous manipulation would further extend this empirical validation to articulated finger motion and more complex contacts. Such an extension would use finger-link point trajectories and adapt the action expert to output hand-joint commands. We leave dedicated dexterous finetuning and evaluation to future work.

Manuscript updates

The revised manuscript includes the following additions.

  • Q1 & W1 — Direct consistency in Appendix H.1, Table 15.
  • Q2 & W1 — Human-data ablation in Appendix H.5, Table 19, and hand/gripper inputs in Appendix H.4.
  • Q3 & W2 — Hardware conditions in Section 4.3 and simulation OOD results in Appendix C, Table 11.

References

[1] Physical Intelligence et al. π0.5\pi_{0.5}: A Vision-Language-Action Model with Open-World Generalization. 2025. arXiv:2504.16054

[2] T. Yuan et al. Fast-WAM: Do World Action Models Need Test-Time Future Imagination? 2026. arXiv:2603.16666

Thank you for the additional experiments and clarifications.

Q1. Consistency between action flow and executed gripper motion

The new results demonstrate a very strong correlation between the predicted action flow and the gripper motion observed after execution, which addresses part of my concern. This strong correlation also makes me more interested in how much performance could improve if the action expert were allowed to attend to the predicted flow. Although this could introduce additional inference latency, it might also yield substantial performance gains. I believe this is a worthwhile direction for further exploration. However, I do not require this experiment during the rebuttal period, and this concern is not a decisive factor in my final assessment.

Q2. Contribution of human demonstrations

Could the authors explain why adding human demonstrations improves in-distribution performance? My understanding is that human demonstrations do not provide gripper data as precise as those available in robot demonstrations. It would therefore be helpful to explain what additional information the human data contribute and why this benefits performance even in the ID setting.

Furthermore, do human demonstrations provide additional gains in OOD performance? My understanding is that the reported ablation results were obtained by fine-tuning on the full RoboTwin dataset and then evaluating in distribution. Please clarify whether this interpretation is correct. An OOD comparison between robot-only and robot-plus-human pretraining would help establish whether human demonstrations offer additional generalization benefits.

Q3. Real-world evaluation and OOD robustness

The real-world evaluation remains too limited in both the number of trials and the range of OOD conditions to convincingly establish the method’s advantages on real robots. In particular, for a 3D-native policy such as UNITAS, it would be valuable to evaluate OOD perturbations that strongly affect 2D observations while leaving the underlying 3D geometry largely unchanged. Such experiments could more directly demonstrate the advantages of UNITAS over conventional 2D-based models and provide strong support for the effectiveness of its 3D formulation.

Given the limited hardware evaluation, I remain uncertain about the method’s real-world performance. Could the authors explain what prevented a broader set of real-world experiments? Was data collection a major difficulty? If so, could this indicate significant practical limitations in collecting the required data and deploying the method on real robots?


Overall, these concerns have not been sufficiently addressed. Given that my initial rating was already relatively positive, I will maintain my current rating.

AuthorsOpenReview ↗

Response to Reviewer LuaC

Q1. Conditioning the action expert on predicted flow

Thank you for raising this very interesting and valuable question. In fact, we explored this design during model development and found that allowing the action expert to attend to action-flow tokens decreased the success rate on RoboTwin from 91.1% to 87.2%. One possible explanation is that the strong correlation between action flow and commands encourages the action expert to learn a shortcut, decoding commands primarily from flow tokens and reducing its reliance on observation and language context. Since flow tokens are compressed representations produced by our trajectory tokenizer, whose reconstruction errors may affect precise control. At inference, errors in generated flow can further propagate into command predictions.

These observations motivated our current attention-mask design. Action flow, scene flow, and observation–language context are processed by a shared backbone, allowing joint dynamics training to shape the contextual representations. The action expert attends to these learned context features and predicts executable commands through its dedicated branch. This provides a pathway for transferring dynamics knowledge to control while preserving a dedicated command-prediction objective.

We appreciate the opportunity to share the reasoning behind this design, and will add this discussion to the revised manuscript.

Q2. Contribution of human demonstrations

Your interpretation is correct. Both variants were fine-tuned on the full RoboTwin downstream training dataset and evaluated in distribution.

Our shared action-flow representation allows human demonstrations to provide additional motion targets, helping align language and observations with task-relevant 3D motion during pretraining. Despite differences in trajectory precision and hand geometry, this richer supervision can provide a better initialization for downstream robot fine-tuning and improve performance even under ID evaluation. These gains are also consistent with beneficial cross-embodiment transfer through the shared action-flow representation.

Q3. Additional real-world OOD evaluation

Following your suggestion, we conducted additional tests on Organize Books with paired robot-height and head-pitch angle perturbations of (+10 cm, −5°) and (−10 cm, +5°). Height is measured relative to the table, and the camera-angle adjustments keep the task objects within the field of view. UNITAS uses the previously trained checkpoint. Each method was evaluated over 20 trials per perturbed condition.

MethodOriginal configuration+10 cm, −5°−10 cm, +5°
Fast-WAM13/20 (65%)6/20 (30%)4/20 (20%)
π0.5\pi_{0.5}15/20 (75%)10/20 (50%)9/20 (45%)
UNITAS18/20 (90%)17/20 (85%)16/20 (80%)

UNITAS maintains success rates of 85% and 80% under the two perturbations. Its average decrease from the original setting is 7.5 percentage points, compared with 27.5 points for π0.5\pi_{0.5} and 40 points for Fast-WAM. These results provide additional real-world evidence of robustness under the tested configuration changes, consistent with our simulation findings.


Thank you again for your thoughtful feedback! We hope that you could raise your rating if your concerns have been adressed.

Reviewer_HJGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

UNITAS proposes a world–action model that aligns visual observations, robot gripper or human hand motion, and scene dynamics in a shared metric 3D coordinate frame. Its main components are world-aligned visual positional embeddings, a physical-time trajectory tokenizer, and coupled experts trained to predict action flow, scene flow, and executable robot commands. The model supports direct policy execution, which omits future-flow generation, and action-conditioned scene prediction. It is trained using a mixture of simulation, real-robot, and human hand demonstrations. The paper reports strong manipulation results across several simulation benchmarks, improved robustness to viewpoint changes, and lower scene-trajectory prediction errors than the compared methods. Real-world policy evaluation consists of two manipulation tasks with 20 trials per method per task.

Critical strengths
  1. The shared metric 3D representation is well motivated and supported by the viewpoint-robustness results.

Aligning observations, gripper/hand trajectories, and scene motion in a common coordinate frame is a sensible inductive bias for manipulation under camera-viewpoint changes. The ablation in Table 3 directly supports this motivation: removing visual 3D positional encoding reduces success under camera perturbations from 79.11% to 66.73%. This connection between the representation design and the measured robustness is a consequential strength of the paper.

  1. The implementation is reasonable and practical, with substantial simulation validation.

The physical-time trajectory tokenizer provides a coherent interface for motion data with different sampling rates, while the coupled-expert architecture permits efficient policy execution without requiring future-flow generation at deployment. The evaluation covers both manipulation success and action-conditioned 3D prediction, including a direct comparison with PointWorld and the additional action–scene consistency experiment in Appendix F.2. These results provide useful evidence that the proposed representation can support both control and dynamics prediction.

Critical weaknesses
  1. The scope and additional downstream value of the world-model component are not sufficiently established.

The scene representation predicts trajectories of initially observed points, and Appendix G.1 explicitly states that unobserved surfaces are not synthesized. This does not, by itself, invalidate the world-model terminology. However, it leaves unclear how the model handles newly exposed geometry, contacts involving initially unobserved surfaces, or extended interactions in simulator mode.

More importantly, policy-only inference omits both future action-flow and scene-flow generation. Table 3 shows a 0.73 percentage-point improvement from adding scene-flow supervision, whereas action-flow supervision and motion history have substantially larger effects. The evidence therefore leaves the distinctive contribution of scene-dynamics modeling to policy learning insufficiently isolated.

The standard rectified-flow objectives are a reasonable choice, but the loss formulation does not explicitly enforce kinematic agreement between the separately generated action flow and executable commands in Eq. (3). Appendix F.2 provides useful indirect evidence through their resulting scene effects, yet does not directly measure this agreement or demonstrate that consulting predicted scene dynamics improves action selection. Overall, the paper should more clearly distinguish the capabilities demonstrated by auxiliary supervision and conditional point-trajectory prediction from any broader downstream role of the generated 3D dynamics.

  1. The evidence for embodiment generality and real-world applicability is limited.

The explicit motion representation uses points sampled from predefined gripper geometry for robots and MANO-derived hand landmarks for human demonstrations, while executable outputs remain end-effector pose and gripper commands. Although URDFs and joint states are used in preprocessing, and robot morphology may also be visible in the observations, the paper does not establish how the common representation accounts for embodiment-dependent effects involving the rest of the robot. For example, similar gripper trajectories can have different feasibility and collision implications under different arm configurations or morphologies.

The paper also lacks a controlled ablation isolating the contribution of human data and an evaluation that specifically tests transfer to held-out embodiments. Including heterogeneous demonstrations in pretraining establishes compatibility with multiple data sources, but does not by itself establish generalization across substantially different embodiments. Finally, the real-world study is restricted to two tasks with 20 trials each. These experiments demonstrate feasibility, but provide limited evidence for the broader transfer and practical robustness suggested by the unified formulation.

Questions for authors
  1. World-model scope — clarification.

Appendix G.1 states that unobserved surfaces are not synthesized. How are newly visible surfaces, changing query sets, and interactions involving initially unobserved geometry handled across successive predictions in simulator mode? Is this mode intended to predict only the trajectories of the initially observed points over a finite prediction window?

  1. Downstream contribution — clarification and evidence relevant to the current claims.

Beyond auxiliary supervision and the reported prediction benchmarks, are generated scene trajectories used for action selection, replanning, or policy improvement anywhere in the evaluated system? For the 0.73 percentage-point gain in Table 3, can the authors provide variability estimates or additional matched-setting results that establish the contribution of scene-flow supervision?

  1. Embodiment information and transfer — clarification and evidence relevant to the current claims.

What information about morphology and configuration outside the hand/gripper is explicitly available to the model, rather than used only in preprocessing? For human demonstrations, is the representation limited to the stated MANO-derived hand landmarks, or is additional body information provided? Are controlled human-data ablations or held-out-embodiment results available that distinguish a common input representation from demonstrated embodiment transfer?

6 comments
AuthorsOpenReview ↗

Response to Reviewer HJGb (Part I)

W1 & Q1. Scope of simulator-mode prediction

Thanks for your insightful questions. Each finite prediction window tracks only the points visible in its initial observation, preserving their identities throughout the window. When a new RGB-D observation becomes available, the next prediction can incorporate newly visible surfaces through a refreshed query set. This scope reflects partial observability: hidden geometry, such as the contents of a closed drawer, cannot be uniquely determined from the initial observation alone. PointWorld [1] and ParticleFormer [2] use a similar observation-grounded modeling approach. The same ambiguity applies to 2D world-action models: they may generate plausible unseen content using learned priors, but this content need not match the actual hidden scene.

W1 & Q2. Downstream value of scene prediction and action consistency

We appreciate the reviewer’s request for clearer evidence of the downstream value of scene prediction and action–execution consistency. The original policy-only results assess the training-time contribution of scene supervision, without generating scene flow at inference. We now provide three additional evaluations.

1. Training-time scene supervision

To accelerate training and reduce GPU memory usage, our default setting retains only 1% of scene points for supervision (r=0.99r=0.99) in the maniscript. This sparse supervision may partly explain the modest LIBERO-Plus gain from 85.74% to 86.47% (Section 4.4, Table 3).

We added a RoboTwin ablation comparing no scene supervision with drop rates of 0.99, 0.90, and 0.75, retaining 1%, 10%, and 25% of scene points, respectively. All configurations are evaluated at their 60k-iteration checkpoints. The main RoboTwin results in Table 1 use 200k-iteration checkpoints, however, extending all ablation runs to 200k iterations was not feasible within the available computational budget and rebuttal period. Trajectory and position errors are in mm, rotation errors in degrees, and success rates in percent.

ConfigurationAction-flow mean ↓Action-flow final ↓EEF position mean ↓EEF position final ↓EEF rotation mean ↓Closed-loop success ↑
No scene supervision8.8512.734.818.301.8274.6
Drop rate 0.995.308.274.267.281.4278.3
Drop rate 0.905.869.054.357.241.3979.2
Drop rate 0.755.549.484.437.401.4178.9

We make the following observations. Scene supervision reduces all reported prediction errors and improves closed-loop success by 3.7–4.6 percentage points across the tested drop rates. Retaining 10% or 25% of scene points provides additional success gains of 0.9 and 0.6 percentage points over the default 1% setting.

2. Inference-time action selection

We independently sample K=4K=4 command chunks and predict scene flow under each candidate’s associated action-flow condition. We fit rigid object poses to the predicted points and render the available simulator meshes at those poses. Qwen3-VL-4B-Instruct [3] evaluates these rendered outcomes against the current observation and instruction to select a command chunk. After execution, the procedure repeats with the updated observation.

With 100 trials per task and method, this scene-guided selection procedure improves success on all three tasks, increasing mean success from 71.67% to 79.67% (+8 percentage points).

RoboTwin taskDirect policy (%)Scene-guided best-of-4 (%)Gain (pp)
hanging_mug6174+13
put_object_cabinet7782+5
place_can_basket7783+6
Mean over all 3 tasks71.6779.67+8.00
AuthorsOpenReview ↗

Response to Reviewer HJGb (Part II)

3. Direct agreement between action flow and executed motion

Although the loss does not explicitly enforce kinematic agreement, the two branches share context and are trained against corresponding demonstration targets. We directly evaluate their resulting agreement during closed-loop execution.

We execute the generated EEF action commands and reconstruct gripper trajectories from the actual joint states using URDF forward kinematics. We compare these with predicted action flow in the same coordinate frame, matching point identities and executed steps. Correlations use displacements relative to their common initial positions.

BenchmarkCCC ↑Pearson ↑ADE ↓ (mm)FDE ↓ (mm)
RoboTwin, 50 tasks0.94660.95119.4116.53
LIBERO0.98320.98624.926.84

ADE measures mean trajectory error, and FDE measures error at the last executed step of each prediction window. Pearson measures displacement correlation, while CCC additionally accounts for differences in mean and scale. Across both benchmarks, Pearson correlations exceed 95% and ADE remains below 1 cm, indicating close agreement between predicted action flow and executed motion.

W2 & Q3. Embodiment information and transfer

We appreciate the reviewer’s questions about embodiment information and transfer. Motion inputs are limited to 20 points per robot gripper or 21 MANO-derived landmarks per human hand; command prediction also uses the current end-effector state. URDFs and joint states are used in preprocessing to construct trajectories and masks, while neither URDFs nor 3D meshes are fed into the backbone. The shared representation captures hand/gripper motion. Arm or body geometry may be visible in images, but whole-arm reachability and collision constraints remain embodiment-dependent.

To establish transfer beyond compatibility with heterogeneous training data, we compare UNITAS with and without multi-embodiment pretraining on the held-out ARX embodiment, excluding ARX data from all training stages. To assess the contribution of human demonstrations, we additionally compare robot-only and robot-and-human pretraining under matched pretraining and fine-tuning budgets.

1. Held-out-embodiment transfer

We compare UNITAS trained from scratch on AgileX RoboTwin with UNITAS pretrained on multiple embodiments and then fine-tuned on AgileX RoboTwin. ARX data are excluded from all training stages for every model. We evaluate five ARX tasks with 20 trials per task per model.

ARX taskX-VLA (%)UNITAS (w/o Pretrain) (%)UNITAS (%)
beat_block_hammer101060
click_alarmclock606570
grab_roller256065
open_laptop107080
place_a2b_left152540
Mean over all 5 evaluated tasks244663

Pretrained UNITAS achieves 63% mean success, exceeding UNITAS without pretraining by 17 percentage points and X-VLA by 39 points, with gains on all five tasks. These results support transfer from multi-embodiment pretraining to the held-out ARX embodiment.

2. Human-data ablation

Under matched pretraining and fine-tuning budgets, adding human demonstrations improves RoboTwin success by 0.61 percentage points on Demo-Clean and 1.06 points on Demo-Randomized.

Pretraining dataRoboTwin Demo-Clean success (%)RoboTwin Demo-Randomized success (%)
Robot only92.3391.98
Robot + human92.9493.04
Gain from adding human data (pp)+0.61+1.06

W2. Real-world applicability and evaluation scope

Hardware evaluation covers two tasks with 20 trials per method per task, varying object positions with a fixed camera. We have clarified this evaluation scope. Controlled camera and layout perturbations are tested in LIBERO-Plus (Appendix C, Table 11).

AuthorsOpenReview ↗

Response to Reviewer HJGb (Part III)

Manuscript updates

The revised manuscript includes the following additions.

  • W1 & Q1 — Prediction scope and related work in Appendix G.1.
  • W1 & Q2 — Point subsampling in Section 3.4 and the supervision ablation in Appendix H.2, Table 16. Action selection in Section 4.5, Figure 6 and Table 5, and Appendix H.3.
  • W2 & Q3 — ARX transfer and embodiment inputs in Appendix H.4, Table 18. Human-data ablation in Appendix H.5, Table 19.
  • W1 — Direct action–execution agreement in Appendix H.1, Table 15.
  • W2 — Hardware conditions in Section 4.3.

References

[1] W. Huang et al. PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation. 2026.

[2] S. Huang et al. ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation. 2025.

[3] S. Bai et al. Qwen3-VL Technical Report. 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Dear Reviewers,

We highly appreciate the constructive and professional comments provided by the reviewers, which has significantly improve our manuscript in every way. The revisions made are marked with blue\textcolor{blue}{\text{blue}} in the revised paper. Below, we outline the specific revisions:

  1. Direct action–execution consistency. We directly compare predicted action flow with gripper trajectories reconstructed from actual joint states using URDF forward kinematics. ADE/FDE are 9.41/16.53 mm across 50 RoboTwin tasks and 4.92/6.84 mm on LIBERO. Pearson correlations exceed 0.95 on both benchmarks. We have added the evaluation protocol and full results to Appendix H.1 (Table 15) in the revised manuscript.

  2. Training-time scene supervision. We have added RoboTwin supervision-density ablations to Appendix H.2 Table 16 in the revised manuscript. At matched 60k-iteration checkpoints, scene supervision improves success from 74.6% to 78.3–79.2% and reduces all reported prediction errors. We have also clarified the default point-drop rate of 0.99 after voxelization in Section 3.4, used to accelerate training and reduce GPU memory usage.

  3. Inference-time action selection. We have added the scene-guided best-of-4 selection procedure and results to Section 4.5, Figure 6 and Table 5 in the revised manuscript. Across three RoboTwin tasks with 100 trials per task per method, the full procedure increases mean success from 71.67% to 79.67% (+8 percentage points). Scene prediction remains optional for policy execution.

  4. Held-out-embodiment transfer. We evaluated five ARX tasks with 20 trials per task per model, excluding ARX data from all training stages. Mean success is 24% for X-VLA, 46% for UNITAS without pretraining, and 63% for UNITAS with multi-embodiment pretraining. We have added the evaluation protocol and results to Appendix H.4 Table 18 in the revised manuscript.

  5. Contribution of human demonstrations. Under matched pretraining and fine-tuning budgets, adding human demonstrations improves RoboTwin success by 0.61 percentage points on Demo-Clean and 1.06 points on Demo-Randomized. We have added this comparison to Appendix H.5 Table 19 in the revised manuscript.

  6. Clarifications and corrections. In the revised manuscript, we have clarified the prediction scope and related work in Appendix G.1, embodiment inputs in Appendix H.4, and hardware evaluation conditions in Section 4.3. We have also corrected the Pack the Ball success rate to 80% (16/20) in Section 1.


Finally, we highly appreciate the constructive comments given by all reviewers again. We are still eager to hear more comments and recommendations to improve this work. These constructive comments will help us further strengthen this work.

Best regards,
The Authors

AuthorsOpenReview ↗

Willing to Address Additional Comments

Dear reviewers,

We would like to highly appreciate all reviewers' efforts and time again in providing valuable comments and constructive suggestions for improvement of our submission. We hope that the clarifications and additional evaluations provided in the responses have addressed all reviewers' questions and concerns.

We are always ready to provide additional clarifications should you have any questions and concerns during the discussion period, due on 18th Nov, only two days remain.

Thank you very much!

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.