- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is motivated by the difficulty of training robot world models to predict diverse interactions outcomes from datasets dominated by successful demonstrations. HF-WM combines video generation with stabilized pointflow supervision to learn from human interaction videos while addressing embodiment differences and camera motion. It then applies GRPO to perturbed actions, using action alignment and physical plausibility rewards to improve failure modeling without paired failure videos. Experiments on successful and failed robot trajectories show improved video quality and action following. Evaluation across six robot policies also shows close agreement between predicted and real world success rates and rankings.
Critical strengths
Improving world models' predictions of unsuccessful interactions is an important research problem. Perturbing successful demonstrations and applying reward-guided post-training offers an interesting way to address it without real failure videos.
Critical weaknesses
-
The paper motivates its approach by using human interaction data and failure-aware training to improve interaction dynamics modeling. Although the experiments show gains, this could come from better prediction of the robot's own motion or its effects on objects or the environment. Predicting the robot's motion is a more constrained problem because it is already largely specified by the input skeleton sequence, and additional data for this purpose should be relatively straightforward to collect or synthesize. PPredicting how objects respond to robot actions is more important for robotic manipulation and provides a stronger motivation for learning from human interactions. However, the current quantitative metrics and ablations do not distinguish how much improvement occurs in this capability, which is my main concern. The downstream experiment in the appendix (figure 12) provides some supporting evidence, but the improvement is also incremental.
-
Figure 5 shows a good correlation between predicted and real world policy success rates, but no comparisons with other world models. It is therefore unclear whether HF-WM offers an improvement over existing methods as a policy evaluator.
Questions for authors
- Could the authors clarify whether the actions used during training represent commands sent to the robot or the robot's actual executed trajectories?
- Figure 9 requires geometric perturbations to preserve the original contact, grasp and release events. Since translating or rotating a trajectory can change whether contact occurs, this requirement might penalize valid failure predictions. Could the author clarify this design?
5 comments
W1: Object-Centric Evaluation of Interaction Dynamics
We thank the reviewer for raising this important question. An action-conditioned world model should predict both commanded robot motion and its effects on manipulated objects and the environment. The existing IDM metric mainly measures action consistency, while full-frame reconstruction metrics do not isolate object-level appearance and dynamics. We therefore report an object-centric evaluation of object appearance, spatial state, and motion.
Evaluation protocol. For region-specific evaluation, we use GPT-5 for initial localization and SAM3 for frame-level tracking to obtain manipulated-object and robot-arm masks exclusively from the ground-truth videos. We apply the same masks to every method without re-detecting or re-segmenting generated outputs. We report PSNR and SSIM for the manipulated object, robot arm, and full frame. For the object region, N-EPE measures RAFT optical-flow endpoint error within ground-truth object-motion regions; lower is better. For the robot arm, we report the Franka-specific IDM error defined in the main paper.
Table Manipulated-object, robot-arm, and full-frame evaluation. Region-specific metrics are computed using masks derived from ground-truth videos.
| Method | Manipulated-object PSNR | Manipulated-object SSIM | Manipulated-object N-EPE | Robot-arm PSNR | Robot-arm SSIM | Robot-arm IDM | Full-frame PSNR | Full-frame SSIM | Full-frame LPIPS |
|---|---|---|---|---|---|---|---|---|---|
| OSCAR | 25.562 | 0.848 | 0.00117 | 14.321 | 0.4800 | 0.579 | 23.357 | 0.840 | 0.142 |
| Kinema4D | 23.723 | 0.825 | 0.00135 | 12.069 | 0.3411 | 0.914 | 20.015 | 0.804 | 0.157 |
| Cosmos-Predict2.5 | 23.302 | 0.722 | 0.00181 | 11.564 | 0.3967 | 1.268 | 18.874 | 0.746 | 0.266 |
| Ctrl-World | 21.797 | 0.607 | 0.00116 | 14.074 | 0.4402 | 0.949 | 21.683 | 0.770 | 0.305 |
| Cosmos3 | 19.776 | 0.552 | 0.00144 | 11.421 | 0.3834 | 1.108 | 20.233 | 0.766 | 0.178 |
| TesserAct | 12.617 | 0.476 | 0.00495 | 10.043 | 0.3184 | 1.663 | 17.383 | 0.757 | 0.263 |
| Ablations | |||||||||
| HF-WM w/o human data | 24.185 | 0.787 | 0.000911 | 15.083 | 0.5369 | 0.406 | 21.573 | 0.835 | 0.117 |
| HF-WM (SFT-only) | 27.196 | 0.881 | 0.000888 | 15.107 | 0.5516 | 0.355 | 25.828 | 0.887 | 0.086 |
| HF-WM (with GRPO) | 28.418 | 0.894 | 0.000854 | 15.676 | 0.5536 | 0.332 | 26.077 | 0.889 | 0.082 |
Results. Both HF-WM (SFT-only) and HF-WM (with GRPO) outperform all six competing methods across all three manipulated-object metrics, and also lead in robot-arm reconstruction, action fidelity, and full-frame quality. More importantly, the ablations directly validate our two core design choices. The consistent gains from incorporating human interaction data demonstrate that our human-to-robot learning framework effectively transfers interaction knowledge beyond robot-only training, improving the prediction of object-state transitions and motion dynamics. GRPO further improves every reported metric over SFT-only, showing that failure-aware post-training strengthens not only action adherence but also the modeling of interaction-induced physical consequences. Together, these broad and complementary gains establish HF-WM's clear advantage at the manipulated-object, robot-arm, and full-frame levels, demonstrating that it more faithfully predicts both commanded robot motion and its downstream effects on the environment.
We would also like to clarify that Figure 12 in the appendix is a supplementary experiment intended to demonstrate that HF-WM's predictions can support test-time action selection and improve closed-loop performance. In our LIBERO-90 setup, this pipeline increases the success rate from 27.78% to 30.0% without updating the base policy. The low baseline success rate indicates that these tasks remain challenging for the evaluated policy. Although HF-WM predicts the outcomes of candidate actions, the benefits of selection also depend on the quality of the candidates generated by and the accuracy of RoboMeter's scoring. The magnitude of this end-to-end gain therefore should not be treated as a direct measure of HF-WM's predictive capability.
Our primary downstream validation is policy evaluation, which more directly assesses how accurately HF-WM predicts real-world policy performance. As shown in the table accompanying our response to W2, under the same evaluation protocol, HF-WM achieves a Pearson correlation of 0.983, compared with 0.345 for OSCAR and 0.142 for Ctrl-World. Its success-rate MAE is 2.41 percentage points, representing reductions of approximately 31.6% and 74.5%, respectively. These comparisons provide direct evidence of HF-WM's effectiveness in our primary downstream setting and its consistent advantages over the evaluated baselines.
W2: Additional Comparison Experiments on Downstream Tasks
We thank the reviewer for this suggestion. To further strengthen the comparison on downstream tasks, we selected OSCAR and Ctrl-World, two relatively strong and representative baselines from Table 5. Following exactly the same evaluation protocol as HF-WM, we evaluated each model on the same 90 RoboArena samples across six robot policies, resulting in rollouts per model. We report the task success rate for each policy and compute both Pearson's correlation coefficient and success-rate MAE (in percentage points) between the six-dimensional policy-level success-rate vector predicted by each world model and its real-world counterpart. As shown in the table, HF-WM achieves both the highest Pearson correlation () and the lowest success-rate MAE ( percentage points), outperforming OSCAR (, points) and Ctrl-World (, points). Although OSCAR obtains relatively high absolute success rates for several policies, its cross-policy performance trend is less consistent with the real-world results. Ctrl-World, meanwhile, substantially underestimates the overall task success rates. These results demonstrate that HF-WM more faithfully captures both the relative performance trend across robot policies and their absolute real-world success rates.
Table Comparison with additional world-model baselines on RoboArena. Each model is evaluated on the same 90 samples across six robot policies, resulting in 540 rollouts per model. For each world model, the value in parentheses is the signed deviation from the corresponding real-world result in percentage points. Negative and positive deviations denote underestimation and overestimation, respectively, and smaller absolute deviations are better.
| Model | -FAST success rate (%) | PG-FAST success rate (%) | PG-FAST+ success rate (%) | PG-FSQ success rate (%) | PG-flow success rate (%) | -flow success rate (%) | Pearson's | Success-rate MAE (pp) |
|---|---|---|---|---|---|---|---|---|
| Real World | 20.0 | 11.1 | 13.3 | 10.0 | 10.0 | 4.4 | -- | -- |
| OSCAR | 13.3 (-6.7) | 10.0 (-1.1) | 11.1 (-2.2) | 10.0 (0.0) | 14.4 (+4.4) | 11.1 (+6.7) | 0.345 | 3.52 |
| Ctrl-World | 2.2 (-17.8) | 2.2 (-8.9) | 2.2 (-11.1) | 1.1 (-8.9) | 2.2 (-7.8) | 2.2 (-2.2) | 0.142 | 9.44 |
| HF-WM (Ours) | 15.6 (-4.4) | 8.9 (-2.2) | 11.1 (-2.2) | 8.9 (-1.1) | 6.7 (-3.3) | 3.3 (-1.1) | 0.983 | 2.41 |
Q1: Clarification on Action Conditioning in HF-WM
We thank the reviewer for this question. HF-WM is an action-conditioned world model that predicts future robot motion and scene dynamics from an initial observation and a numerical action sequence. Specifically, the numerical actions are converted into 2D skeletons using URDF-based forward kinematics and camera calibration parameters, which then condition future video generation. The input actions represent commanded motions rather than measured executed trajectories. The conditioning skeletons are computed entirely from numerical actions, without using any information from future video frames. The recorded future videos serve only as training supervision. While the action skeletons provide spatial constraints on robot motion, they do not directly specify its visual realization. The model must still synthesize temporally coherent robot motion, including appearance, pose changes, and occlusions, while ensuring consistency with the given actions.
Q2: Clarification on Interaction Preservation under Geometric Perturbations
We thank the reviewer for pointing out this concern. We agree that translating or rotating the end-effector trajectory may alter the original contact relationships. To mitigate this issue, we do not apply geometric perturbations uniformly across all trajectories. Instead, we determine the perturbation type and timing based on the gripper event patterns in the original trajectories, aiming to avoid interfering with critical interaction events. Specifically, as shown in Table 4 of the main paper, we first categorize trajectories into seven classes based on gripper-state transitions. For the four classes with explicit gripper transitions, we schedule geometric perturbations after the final gripper-state transition to avoid introducing spatial deviations during critical grasping and release phases. Perturbations are applied after the closing event for Close hold (Open→Closed), after reopening for Close reopen (Open→Closed→Open), after opening for Open hold (Closed→Open), and after the final closing event for Open close (Closed→Open→Closed). This temporal scheduling preserves the key gripper events in the original trajectories, while applying geometric perturbations primarily to the subsequent motion segments. For the remaining three classes without explicit gripper-state transitions, namely Remain open, Partial hold, and Remain closed, there are no explicit grasping or release transitions that require temporal protection. Therefore, geometric perturbations can be applied throughout the motion window while keeping the original gripper-state sequence unchanged. This design is also reflected in the sampling strategy in Table 4. For the four classes with explicit gripper events, we sample unperturbed actions, gripper perturbations, and geometric perturbations with probabilities of 50%, 35%, and 15%, respectively. For the other three classes, we do not apply gripper perturbations and increase the geometric perturbation probability to 50%. The rationale is that, for trajectories with explicit interaction events, we primarily use gripper perturbations to modify interaction conditions and explore alternative object interaction outcomes, while geometric perturbations are used to study spatial motion variations while preserving the original interaction process. For trajectories without explicit gripper events, geometric perturbations are instead used to increase spatial motion diversity. Thus, the two perturbation types serve complementary purposes: geometric perturbations are scheduled to avoid critical gripper events and preserve the original interaction process, whereas gripper perturbations directly modify interaction conditions to explore counterfactual outcomes of grasping, holding, and releasing. We will clarify this design and its motivation in the revised manuscript.